search

pepkit / eido

Validator for PEP objects
http://eido.databio.org
BSD 2-Clause "Simplified" License
4 stars 6 forks source link

RemoteYAMLError #52

Closed KuechlerO closed 1 year ago

KuechlerO commented 1 year ago

Hello guys,

firstly thx for your project and the efforts you pushed into it :)

After weeks of utilizing PEP without any issues I encountered this error the last days:

RemoteYAMLError in line 3 of /sc-scratch/sc-scratch-btg/olik_splicing_project/splice-prediction/snakemake_workflows/Snakemake_Main/Snakefile:
Could not load remote file: http://schema.databio.org/pep/2.1.0.yaml. Original exception: <HTTPError 403: 'Forbidden'>
  File "/sc-scratch/sc-scratch-btg/olik_splicing_project/splice-prediction/snakemake_workflows/Snakemake_Main/Snakefile", line 3, in <module>
  File "/home/kuechleo/mambaforge/envs/snakemake/lib/python3.10/site-packages/eido/validation.py", line 50, in validate_project
  File "/home/kuechleo/mambaforge/envs/snakemake/lib/python3.10/site-packages/eido/schema.py", line 76, in read_schema
  File "/home/kuechleo/mambaforge/envs/snakemake/lib/python3.10/site-packages/eido/schema.py", line 68, in _recursively_read_schemas
  File "/home/kuechleo/mambaforge/envs/snakemake/lib/python3.10/site-packages/eido/schema.py", line 76, in read_schema
  File "/home/kuechleo/mambaforge/envs/snakemake/lib/python3.10/site-packages/peppy/utils.py", line 122, in load_yaml

At first I thought it could be the settings of the cluster, where I run the pipeline, but even on my home computer the same error gets produced. Sanity check with wget: wget http://schema.databio.org/pep/2.1.0.yaml runs without any problems. But python:

from urllib.request import urlopen
urlopen("http://schema.databio.org/pep/2.1.0.yaml")

returns urllib.error.HTTPError: HTTP Error 403: Forbidden.

Do you have ideas for the potential reason?

Also: Does there exist a possibility to use a local pep-version-config file? So that instead of pep_version: 2.1.0 in the pep-configuration file, I can directly reference a local file?

My set-up:

KuechlerO commented 1 year ago

Also update to eido=0.1.9 and peppy=0.35.4 did not solve the issue.

nsheff commented 1 year ago

It looks like github is denying the request because it's coming from Python...

https://stackoverflow.com/questions/16627227/problem-http-error-403-in-python-3-web-scraping

I am also getting the same response here. We may need to change the way we request remote files in peppy

You can reference the file locally, and it should work. Can you give that a try and see if it solves it in the short term?

KuechlerO commented 1 year ago

Ah, I see. How can I reference the file locally? I was not able to find any documentation on it...

nsheff commented 1 year ago

I figured out what this is. I had recently change the DNS settings for schema.databio.org, which I changed to being proxied by cloudflare. Cloudflare was blocking the automated requests from Python.

I've reverted back to the old non-proxied DNS settings, and this solved the issue for me. Check now and see if things work for you. If not, wait 24 hours for DNS settings to reset and try again. Let me know if you have further issues.

KuechlerO commented 1 year ago

Yes, now it works again. Thank you very much! :)

Still, could you please point out to me, how I can set up a local reference in order to avoid such a problem in the future?

KuechlerO commented 1 year ago

Yes, now it works again. Thank you very much! :)

Still, could you please point out to me, how I can set up a local reference in order to avoid such a problem in the future?

I figured it out! Thanks again for your quick fix and the amazing work 👍