search

baliga-lab / cmonkey2

Python port of cMonkey, a machine-learning based method for clustering
GNU Lesser General Public License v3.0
26 stars 16 forks source link

urllib2.HTTPError: HTTP Error 404: Not Found #62

Closed ABanaeiEsfahani closed 7 years ago

ABanaeiEsfahani commented 7 years ago

Hello,

I had run cMonkey on my dataset while ago without any problem. But seems like I can't manage manage to run it again with slightly different parameters. Could you please help me? Thanks.

Best regards, Amir

cmonkey2 ./Mtb/WayneAndTbX007_MedianNormalization_RatioByMeanD00.tsv --organism mtu --string ./Mtb/83332_v10.tsv --rsat_base_url http://pedagogix-tagc.univ-mrs.fr/rsat --rsat_organism Mycobacterium_tuberculosis_H37Rv_uid57777 --rsat_features gene --out ./Output_files --use_BSCM --numclusters 600 2016-11-15 16:14:58 INFO checking MEME... 2016-11-15 16:14:59 INFO Input matrix has # rows: 2683, # columns: 33 2016-11-15 16:14:59 INFO # clusters/row: 2 2016-11-15 16:14:59 INFO # clusters/column: 400 2016-11-15 16:14:59 INFO # CLUSTERS: 600 2016-11-15 16:14:59 INFO use operons: 1 2016-11-15 16:14:59 INFO using MEME version 4.10.2 2016-11-15 16:15:01 INFO attempting automatic download of operons from Microbes Online 2016-11-15 16:15:01 INFO Loading STRING file at './Mtb/83332_v10.tsv' 2016-11-15 16:15:01 INFO KEGG = 'Mycobacterium tuberculosis H37Rv uid57777' -> RSAT = 'Mycobacterium_tuberculosis_H37Rv_uid57777' 2016-11-15 16:15:01 INFO Creating networks... 2016-11-15 16:15:01 INFO stringdb.read_edges2() 2016-11-15 16:15:05 INFO Finished loading ./Mtb/83332_v10.tsv 2016-11-15 16:15:08 INFO Processing network 5% 2016-11-15 16:15:08 INFO Processing network 10% 2016-11-15 16:15:08 INFO Processing network 15% 2016-11-15 16:15:08 INFO Processing network 20% 2016-11-15 16:15:08 INFO Processing network 25% 2016-11-15 16:15:09 INFO Processing network 30% 2016-11-15 16:15:09 INFO Processing network 35% 2016-11-15 16:15:09 INFO Processing network 40% 2016-11-15 16:15:09 INFO Processing network 45% 2016-11-15 16:15:09 INFO Processing network 50% 2016-11-15 16:15:10 INFO Processing network 55% 2016-11-15 16:15:10 INFO Processing network 60% 2016-11-15 16:15:10 INFO Processing network 65% 2016-11-15 16:15:10 INFO Processing network 70% 2016-11-15 16:15:10 INFO Processing network 75% 2016-11-15 16:15:11 INFO Processing network 80% 2016-11-15 16:15:11 INFO Processing network 85% 2016-11-15 16:15:11 INFO Processing network 90% 2016-11-15 16:15:11 INFO Processing network 95% 2016-11-15 16:15:11 INFO Processing network 100% 2016-11-15 16:15:11 WARNING 15 (out of 1247720) nodes not found in synonyms 2016-11-15 16:15:11 WARNING 1292 (out of 1247720) nodes not found in canonical gene names 2016-11-15 16:15:12 INFO stringdb.read_edges2(), 407198 edges read, 216662 edges ignored 2016-11-15 16:15:14 INFO MicrobesOnline - make_network() 2016-11-15 16:15:14 INFO MicrobesOnline.get_operon_predictions_for(83332) Traceback (most recent call last): File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/bin/cmonkey2", line 36, in cmonkey_run.run() File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/cmonkey_run.py", line 512, in run self.prepare_run() File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/cmonkey_run.py", line 474, in prepare_run thesaurus = self.organism().thesaurus() File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/cmonkey_run.py", line 231, in organism self.organism = self.make_organism() File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/cmonkey_run.py", line 341, in make_organism self['fasta_file']) File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/organism.py", line 244, in init fasta_file) File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/organism.py", line 117, in init OrganismBase.init(self, code, network_factories, ratios=ratios) File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/organism.py", line 72, in init self.networks.append(make_network(self, ratios)) File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/microbes_online.py", line 231, in make_network edges = get_operon_edges(microbes_online, organism) File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/microbes_online.py", line 215, in get_operon_edges __get_predictions(microbes_online, organism), organism) File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/microbes_online.py", line 199, in __get_predictions organism.taxonomy_id()) File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/microbes_online.py", line 54, in get_operon_predictions_for return util.read_url_cached(url, cache_file).decode('utf-8') File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/util.py", line 255, in read_url_cached outfile.write(read_url(url)) File "/IMSB/ra/bamir/Computation_on_Clusters/My_virtual_env/lib/python2.7/site-packages/cmonkey/util.py", line 244, in read_url return urlopen(url).read() File "/cluster/apps/python/2.7.2/x86_64/lib64/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/cluster/apps/python/2.7.2/x86_64/lib64/python2.7/urllib2.py", line 400, in open response = meth(req, response) File "/cluster/apps/python/2.7.2/x86_64/lib64/python2.7/urllib2.py", line 513, in http_response 'http', request, response, code, msg, hdrs) File "/cluster/apps/python/2.7.2/x86_64/lib64/python2.7/urllib2.py", line 438, in error return self._call_chain(args) File "/cluster/apps/python/2.7.2/x86_64/lib64/python2.7/urllib2.py", line 372, in _call_chain result = func(args) File "/cluster/apps/python/2.7.2/x86_64/lib64/python2.7/urllib2.py", line 521, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 404: Not Found

weiju commented 7 years ago

Hi, it looks like microbesonline.org has removed or moved the downloadable files for operon predictions. In your case it would be a file called

http://www.microbesonline.org/operons/gnc83332.named

I have checked and at least right now those files are not available anymore. This file is cached typically stored in the cache directory, so if you happen to have this file from your previous runs, then you should be able to work around this problem by copying it to your cache directory.

If you don't have that file anymore please let me know and I will send you that file.

In the meanwhile, we will contact microbesonline.org to find out what caused this change and provide a workaround.

Thank you very much for bringing this to our attention.

weiju commented 7 years ago

MicrobesOnline fixed the issue and the download of operon predictions should work as before.

ABanaeiEsfahani commented 7 years ago

Thanks a lot :)