search

openvax / pyensembl

Python interface to access reference genome features (such as genes, transcripts, and exons) from Ensembl
Apache License 2.0
365 stars 68 forks source link

Cannot update Ensembl release #269

Open DDudka9 opened 1 year ago

DDudka9 commented 1 year ago

I use pyensembl on daily bases and I love it! However, recently I noticed that I am not able to update Ensembl release from 90 to 107 (latest) for cat (error message attached below). I also see that the assembly that pyensembl is looking for (Felis_catus_6.2) does not match the current Ensembl release (Felis_catus_9.0).

When I try to update to 107:

python -m pip install pyensembl==2.0.1
pyensembl install --species cat --release 107

I get this error:

Downloading ftp://ftp.ensembl.org/pub/release-107/gtf/felis_catus/Felis_catus.Felis_catus_6.2.107.gtf.gz to /Users/damian/Library/Caches/pyensembl/Felis_catus_6.2/ensembl107/Felis_catus.Felis_catus_6.2.107.gtf.gz Traceback (most recent call last): File "/Users/damian/anaconda3/envs/py37/lib/python3.7/urllib/request.py", line 2450, in retrfile self.ftp.cwd(file) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/ftplib.py", line 631, in cwd return self.voidcmd(cmd) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/ftplib.py", line 278, in voidcmd return self.voidresp() File "/Users/damian/anaconda3/envs/py37/lib/python3.7/ftplib.py", line 251, in voidresp resp = self.getresp() File "/Users/damian/anaconda3/envs/py37/lib/python3.7/ftplib.py", line 246, in getresp raise error_perm(resp) ftplib.error_perm: 550 Failed to change directory.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/Users/damian/anaconda3/envs/py37/lib/python3.7/urllib/request.py", line 1573, in ftp_open fp, retrlen = fw.retrfile(file, type) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/urllib/request.py", line 2452, in retrfile raise URLError('ftp error: %r' % reason) from reason urllib.error.URLError: <urlopen error ftp error: error_perm('550 Failed to change directory.')>

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/Users/damian/PycharmProjects/Freeda_pyinstaller_04_29_2022/freeda_pipeline.py", line 375, in codon_frequencies=args.codon_frequencies, excluded_species=args.excluded_species) File "/Users/damian/PycharmProjects/Freeda_pyinstaller_04_29_2022/freeda_pipeline.py", line 203, in freeda_pipeline biotype, all_genes_ensembl = input_extractor.generate_ref_genome_object(wdir, ref_species) File "/Users/damian/PycharmProjects/Freeda_pyinstaller_04_29_2022/freeda/input_extractor.py", line 379, in generate_ref_genome_object ensembl.download() # this is suppose to bypass installing the release from outside python File "/Users/damian/anaconda3/envs/py37/lib/python3.7/site-packages/pyensembl/genome.py", line 266, in download self._set_local_paths(download_if_missing=True, overwrite=overwrite) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/site-packages/pyensembl/genome.py", line 224, in _set_local_paths overwrite=overwrite) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/site-packages/pyensembl/genome.py", line 188, in _get_gtf_path overwrite=overwrite) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/site-packages/pyensembl/genome.py", line 181, in _get_cached_path overwrite=overwrite) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/site-packages/pyensembl/download_cache.py", line 312, in local_path_or_install_error overwrite=overwrite) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/site-packages/pyensembl/download_cache.py", line 288, in download_or_copy_if_necessary overwrite) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/site-packages/pyensembl/download_cache.py", line 234, in _download_if_necessary timeout=3600) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/site-packages/datacache/download.py", line 110, in _download_and_decompress_if_necessary use_wget_if_available=use_wget_if_available) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/site-packages/datacache/download.py", line 68, in _download_to_temp_file download_using_python() File "/Users/damian/anaconda3/envs/py37/lib/python3.7/site-packages/datacache/download.py", line 65, in download_using_python _download(download_url, timeout=timeout)) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/site-packages/datacache/download.py", line 42, in _download response = urllib.request.urlopen(req, data=None, timeout=timeout) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/urllib/request.py", line 525, in open response = self._open(req, data) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/urllib/request.py", line 543, in _open '_open', req) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/urllib/request.py", line 503, in _call_chain result = func(*args) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/urllib/request.py", line 1584, in ftp_open raise exc.with_traceback(sys.exc_info()[2]) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/urllib/request.py", line 1573, in ftp_open fp, retrlen = fw.retrfile(file, type) File "/Users/damian/anaconda3/envs/py37/lib/python3.7/urllib/request.py", line 2452, in retrfile raise URLError('ftp error: %r' % reason) from reason urllib.error.URLError: <urlopen error ftp error: URLError("ftp error: error_perm('550 Failed to change directory.')")>

Thank you for maintaining the package!

iskandr commented 1 year ago

Sorry for the long delay, checking this out now.

iskandr commented 1 year ago

Reproduced on my machine:

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://ftp.ensembl.org/pub/release-107/gtf/felis_catus/Felis_catus.Felis_catus_6.2.107.gtf.gz

Probably there's some new cat reference since release 107, going to check that out on Ensembl now.

iskandr commented 1 year ago

OK, should work as of the code I just pushed: 

➜  ~ pyensembl install --species cat --release 107
2022-12-01 18:07:04,482 - pyensembl.shell - INFO - Running 'install' for EnsemblRelease(release=107, species='felis_catus')
2022-12-01 18:07:04,482 - pyensembl.download_cache - INFO - Fetching /Users/iskander/Library/Caches/pyensembl/Felis_catus_9.0/ensembl107/Felis_catus.Felis_catus_9.0.107.gtf.gz from URL https://ftp.ensembl.org/pub/release-107/gtf/felis_catus/Felis_catus.Felis_catus_9.0.107.gtf.gz

I'll get it on to PyPI later tonight (after unit tests finish on Travis)

DDudka9 commented 1 year ago

Hi Alex,

No worries - thanks for the update. A quick follow up - do you envision in the future adding other model organisms into pyensembl? Such as Drosophila?

Thanks a lot!

Damian

On Dec 1, 2022, at 6:08 PM, Alex Rubinsteyn @.***> wrote:

OK, should work as of the code I just pushed:

➜ ~ pyensembl install --species cat --release 107 2022-12-01 18:07:04,482 - pyensembl.shell - INFO - Running 'install' for EnsemblRelease(release=107, species='felis_catus') 2022-12-01 18:07:04,482 - pyensembl.download_cache - INFO - Fetching /Users/iskander/Library/Caches/pyensembl/Felis_catus_9.0/ensembl107/Felis_catus.Felis_catus_9.0.107.gtf.gz from URL https://ftp.ensembl.org/pub/release-107/gtf/felis_catus/Felis_catus.Felis_catus_9.0.107.gtf.gz I'll get it on to PyPI later tonight (after unit tests finish on Travis)

— Reply to this email directly, view it on GitHub https://github.com/openvax/pyensembl/issues/269#issuecomment-1334562739, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJLCITRNVBEZZZKLAODEL3WLEVWRANCNFSM57QMCXTQ. You are receiving this because you authored the thread.

iskandr commented 1 year ago

Sure! Since they require a little curation to the ftp paths right, I'm doing them one at a time as people request it. Which organisms do you need?

On Thu, Dec 8, 2022, 10:09 AM DDudka9 @.***> wrote:

Hi Alex,

No worries - thanks for the update. A quick follow up - do you envision in the future adding other model organisms into pyensembl? Such as Drosophila?

Thanks a lot!

Damian

On Dec 1, 2022, at 6:08 PM, Alex Rubinsteyn @.***> wrote:

OK, should work as of the code I just pushed:

➜ ~ pyensembl install --species cat --release 107 2022-12-01 18:07:04,482 - pyensembl.shell - INFO - Running 'install' for EnsemblRelease(release=107, species='felis_catus') 2022-12-01 18:07:04,482 - pyensembl.download_cache - INFO - Fetching /Users/iskander/Library/Caches/pyensembl/Felis_catus_9.0/ensembl107/Felis_catus.Felis_catus_9.0.107.gtf.gz from URL https://ftp.ensembl.org/pub/release-107/gtf/felis_catus/Felis_catus.Felis_catus_9.0.107.gtf.gz I'll get it on to PyPI later tonight (after unit tests finish on Travis)

— Reply to this email directly, view it on GitHub < https://github.com/openvax/pyensembl/issues/269#issuecomment-1334562739>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ATJLCITRNVBEZZZKLAODEL3WLEVWRANCNFSM57QMCXTQ . You are receiving this because you authored the thread.

— Reply to this email directly, view it on GitHub https://github.com/openvax/pyensembl/issues/269#issuecomment-1342877106, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAL2OL26AGM3LI5Z4X3CH3WMH23FANCNFSM57QMCXTQ . You are receiving this because you commented.Message ID: @.***>

DDudka9 commented 1 year ago

Nice! I am actually working on a computational pipeline that is used to detect positive selection in model organisms (which I am submitting soon). The pipeline is using pyensembl to generate input for an array of molecular evolution tools. I would be very excited if you could add Drosophila melanogaster and C. elegans. Along these lines, how do I cite pyensembl?

Thanks a lot - I appreciate you maintaining pyensembl - such a useful tool!

Best,

Damian

On Dec 8, 2022, at 10:20 AM, Alex Rubinsteyn @.***> wrote:

Sure! Since they require a little curation to the ftp paths right, I'm doing them one at a time as people request it. Which organisms do you need?

On Thu, Dec 8, 2022, 10:09 AM DDudka9 @.***> wrote:

Hi Alex,

No worries - thanks for the update. A quick follow up - do you envision in the future adding other model organisms into pyensembl? Such as Drosophila?

Thanks a lot!

Damian

On Dec 1, 2022, at 6:08 PM, Alex Rubinsteyn @.***> wrote:

OK, should work as of the code I just pushed:

➜ ~ pyensembl install --species cat --release 107 2022-12-01 18:07:04,482 - pyensembl.shell - INFO - Running 'install' for EnsemblRelease(release=107, species='felis_catus') 2022-12-01 18:07:04,482 - pyensembl.download_cache - INFO - Fetching /Users/iskander/Library/Caches/pyensembl/Felis_catus_9.0/ensembl107/Felis_catus.Felis_catus_9.0.107.gtf.gz from URL https://ftp.ensembl.org/pub/release-107/gtf/felis_catus/Felis_catus.Felis_catus_9.0.107.gtf.gz I'll get it on to PyPI later tonight (after unit tests finish on Travis)

— Reply to this email directly, view it on GitHub < https://github.com/openvax/pyensembl/issues/269#issuecomment-1334562739>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ATJLCITRNVBEZZZKLAODEL3WLEVWRANCNFSM57QMCXTQ . You are receiving this because you authored the thread.

— Reply to this email directly, view it on GitHub https://github.com/openvax/pyensembl/issues/269#issuecomment-1342877106, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAL2OL26AGM3LI5Z4X3CH3WMH23FANCNFSM57QMCXTQ . You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/openvax/pyensembl/issues/269#issuecomment-1342892459, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATJLCIQRL3DFQEPCBSWTP3TWMH4CVANCNFSM57QMCXTQ. You are receiving this because you authored the thread.