Open michaelaye opened 3 years ago
Hi @michaelaye yes, this is a current limitation on the PSA side due to the performance of PDAP with large bundles. I raised a ticket on this internally back in February (PSAPCR-2534 for my own reference), but I'm not sure when it will be fixed. But I'll flag that external users need this addressed!
ah it's the thing you mentioned in Slack already, ok. maybe you could wrap the error message so that it would become clear to the user?
Sure, I would have to check what other conditions may cause the same error type. I suspect that this is just happening due to a database query timing out, and that as a quick fix we could increase the time-out, but this is on the development team side to investigate.
I'm digging a bit into how the HTTP request works, and am wondering if you are sure that this is a server config issue due to large bundles? Because I found this similar issue that states that this behaviour is a bug in Apache 2.4.6 (used currently by the PDAP server) that was fixed in 2.4.13? https://serverfault.com/questions/834193/502-error-reading-from-remote-server-when-upstream-service-errors
The current configured HTTP time-out btw, is unusually short, with 2 seconds:
though I'm not sure how this precisely effects things, because it takes much longer to get the 502 error than 2 seconds?
HI @michaelaye I don't think that HTTP time-out is the issue. I did some timing tests and 68 out of 7813 datasets give this issue, and the largest datasets that I can retrieve have about 50k products (0 products = failed):
Looking at the list that fail, they all look like big datasets (PDS4 bundles like EM16 CaSSIS, ACS, NOMAD, FREND) and large PDS3 datasets (e.g. ROSINA and COSIMA on Rosetta).
So I believe that it is still the dataset size that is the issue, but I will let the archive engineers look into this and provide a solution!
Thx for the package! just in time! :)
Trying to get Cassis labels (all of them). So, following your nice example notebook:
Then i take the ID of the first column and use it with products, but get a proxy error: