fmigneault
commented
1 year ago @tlogan2000 @tlvu
The last error in the notebook displays this link as the cause of error:
https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/datasets/simulations/bias_adjusted/cmip6/pcic/CanDCS-U6/day_BCCAQv2+ANUSPLIN300_MPI-ESM1-2-LR_historical+ssp245_r1i1p1f1_gn_1950-2100.ncml
Which returns the following when I call it manually:
Error {
code = 400;
message = "Unrecognized request";
};This looks like the error I usually get when doing an incorrectly formed request towards THREDDS.
Using the corresponding
https://pavics.ouranos.ca/testthredds/catalog/testdatasets/CanDCS-U6/catalog.html?dataset=testdatasets/CanDCS-U6/day_BCCAQv2%2BANUSPLIN300_MPI-ESM1-2-LR_historical%2Bssp245_r1i1p1f1_gn_1950-2100.ncml
produces the expected results.
I'm guessing there must be a parameter that is incorrectly parsed or forwarded somewhere, which causes the Twitcher redirect to be incorrect. I notice that the testthredds URL encodes the literal + as %2B (as it should, otherwise + = space) when I hover the links in THREDDS. The Twitcher-protected THREDDS using the Catalog links is also valid, and returns the right response, but not with the OpenDAP generated link:
https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/catalog/datasets/simulations/bias_adjusted/cmip6/pcic/CanDCS-U6/catalog.html?dataset=datasets/simulations/bias_adjusted/cmip6/pcic/CanDCS-U6/day_BCCAQv2%2BANUSPLIN300_MPI-ESM1-2-LR_historical%2Bssp245_r1i1p1f1_gn_1950-2100.ncmlOKhttps://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/datasets/simulations/bias_adjusted/cmip6/pcic/CanDCS-U6/day_BCCAQv2+ANUSPLIN300_MPI-ESM1-2-LR_historical+ssp245_r1i1p1f1_gn_1950-2100.ncml.htmlOKhttps://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/datasets/simulations/bias_adjusted/cmip6/pcic/CanDCS-U6/day_BCCAQv2+ANUSPLIN300_MPI-ESM1-2-LR_historical+ssp245_r1i1p1f1_gn_1950-2100.ncml("Data URL" under OpenDAP of previous link) BAD
So, I'm not sure where the notebook retrieves the badly formed link from, without the appropriate URL encoding, and why it doesn't do the same problem using testthredds. Maybe THREDDS internally handles the auto-redirect when it detects literal + (?), but Twitcher doesn't (and shouldn't) do it.
Is it possible to check if some Magpie/Twitcher/Nginx logs are produced on the Ouranos instance when calling the first link (maybe need to start them with DEBUG to getter more descriptive logs)?
tlogan2000
tlvu
Describe the bug
We have noticed intermittant netcdf read errors when accessing opendap links in the PAVICS jupyterhub. Approximate date when the problem began ~ March 01 2023
To Reproduce
The intermittant nature of this problem makes reproducing somewhat difficult but a public notebook on the PAVICS server is available here : https://pavics.ouranos.ca/jupyter/hub/user-redirect/lab/tree/public/logan-public/Tests/THREDDS_Issues_March2023/Random_Thredds_read_errors.ipynb
The notebook will execute a relatively large workflow and uses multiple dask worker processes to accentuate the possibility of a read error
Multiple notebook runs (note only ~5-6 due to time needed) have shown that bypassing twitcher (i.e. thredds with nginx proxy only) always allows successful completion of the calculations whereas accessing opendap links behind the nginx/twitcher combinations typically results in a read error relatively quickly in the workflow.
Although not quantified there also seems to be a general performance hit when accessing data via nginx/twitcher (print outs of execution times in the workflow loop between 25-40 sec with nginx/twitcher versus 18-30 seconds with nginx only). Note also, that the notebook runs 'nginx-only' code first so I do not believe the performance difference is benefiting from caching of data or if it is should benefit the 'twitcher/nginx' run.
Expected behavior
Execution of code without read error