with very large par_dict url, even redirect does no work anymore

[ferrigno]

I do not know if it is related, but I put it here for your investigation.

[ODA][done] jemx_image first requested at 2021-07-01 16:34:41 job_id: 53eeb67f

This request with 500 scws (in a file) contains an URL that gives an error

502 Bad Gateway nginx

URL: https://frontend.staging.odahub.io/cdci/astrooda//dispatch-data/resolve-job-url?job_id=53eeb67f53fa79ac&session_id=O5GH5MDX7TKNII7Q&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJlbWFpbCI6IkNhcmxvLkZlcnJpZ25vQHVuaWdlLmNoIiwibmFtZSI6ImNmZXJyaWdubyIsInJvbGVzIjoiYXV0aGVudGljYXRlZCB1c2VyLCBhZG1pbmlzdHJhdG9yLCBjb250ZW50IG1hbmFnZXIsIGdlbmVyYWwsIGludGVncmFsLXByaXZhdGUtcWxhLCBtYWdpYywgdW5pZ2UtaHBjLWZ1bGwsIHB1YmxpYy1wb29sLWhwYywgYW50YXJlcywgc2RzcyIsImV4cCI6MTYyNTMyMjA1MX0.tRsYlEFE9nBnmtTosDGlc8b0zD-Rnv63-wV6BBMycjc

[volodymyrss]

yes, that's what I meant.
It is possible but not completely straightforward to sort of pass a file through URL. Passing actual list as now is not possible since it is too large.

For now we could add a warning that for request like this, it should be repeated.

API should work though, since it uses POST. Also of course you can repeat the request..

[ferrigno]

API code will contain an outdated token in several cases.

Even after getting an updated token, I get this error ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

However, if the URL does not work, I would avoid putting it in the email

[ferrigno]

I tried to resubmit the query, but it looks stuck

[volodymyrss]

I tried to resubmit the query, but it looks stuck

last thing dispatcher did was to send back large mosaic on 2021-07-02 12:15 how did you submit it? Frontend? Did you reload etc? It was updated.

[ferrigno]

I reloaded the frontend page, logged in with my credentials and made the very exact query (I hope).

The page looks like

[ferrigno]

from a notebook I did

disp = DispatcherAPI(url="http://dispatcher.staging.internal.odahub.io", instrument="mock")

par_dict = { "DEC": "22.01446", "E1_keV": "3", "E2_keV": "20", "RA": "83.633212", "T1": "2016-03-20T10:02", "T2": "2016-03-20T13:02", "T_format": "isot", "detection_threshold": "7", "instrument": "jemx", "integral_data_rights": "public", "jemx_num": "2", "osa_version": "OSA11.1", "product": "jemx_image", "product_type": "Real", "radius": "4", "scw_list": [ "090601130010.001", "090601140010.001", "090701170010.001", "090901150010.001", "091001080010.001", "091200030010.001", "091200040010.001", "091300530010.001", "091700370010.001", "091700380010.001", "091700430010.001", "091700440010.001", "091700450010.001", "091700460010.001", "091700470010.001", "091700480010.001", "091700540010.001", "091800150010.001", "091800160010.001", "091800170010.001", "091800220010.001", "091800230010.001", "091800240010.001", "091800250010.001", "091800260010.001", "091800270010.001", "091800320010.001", "091800330010.001", "091800340010.001", "095800070010.001", "096400060010.001", "096400070010.001", "096500840010.001", "096600070010.001", "097300260010.001", "097300270010.001", "097400700010.001", "097400710010.001", "097501050010.001", "097600150010.001", "097800180010.001", "102100040010.001", "102100050010.001", "102200270010.001", "102300110010.001", "102700040010.001", "102900380010.001", "102900390010.001", "103000900010.001", "103000910010.001", "103100910010.001", "103400570010.001", "103500400010.001", "103600890010.001", "103600900010.001", "103700060010.001", "103800680010.001", "107900180010.001", "107900190010.001", "108100090010.001", "108200850010.001", "108300090010.001", "108300100010.001", "109100670010.001", "109100680010.001", "109200900010.001", "109300280010.001", "109300290010.001", "109400070010.001", "109400070010.002", "109500070010.001", "110100600010.001", "110300070010.001", "110300080010.001", "113700290010.001", "113800370010.001", "113900210010.001", "114000360010.001", "114200040010.001", "114200050010.001", "114301070010.001", "114401040010.001", "114700040010.001", "114900950010.001", "114900960010.001", "114900970010.001", "114900980010.001", "115000300010.001", "115000310010.001", "115000320010.001", "115000560010.001", "115000570010.001", "115000580010.001", "115100140010.001", "115100150010.001", "115100220010.001", "115100230010.001", "115100240010.001", "115100250010.001", "115100400010.001", "115100490010.001", "115100500010.001", "115100740010.001", "115100750010.001", "115100760010.001", "115100770010.001", "115100920010.001", "115101000010.001", "115101010010.001", "115101020010.001", "115101030010.001", "115200140010.001", "115200150010.001", "115200220010.001", "115200230010.001", "115200240010.001", "115200250010.001", "115200400010.001", "115200480010.001", "115200490010.001", "115200500010.001", "115200660010.001", "115200750010.001", "115200760010.001", "115200920010.001", "115201000010.001", "115201010010.001", "115201020010.001", "115300060010.001", "115300070010.001", "115400810010.001", "115400820010.001", "115400840010.001", "115500040010.001", "115500050010.001", "115700210010.001", "115700220010.001", "115700230010.001", "115700580010.001", "115700590010.001", "115700600010.001", "115700610010.001", "115700970010.001", "115700980010.001", "119900290010.001", "120001050010.001", "120201090010.001", "120300840010.001", "120300850010.001", "120300860010.001", "120300880010.001", "120500040010.001", "120500050010.001", "120500060010.001", "120500080010.001", "120700050010.001", "120700060010.001", "120700070010.001", "120700080010.001", "120700090010.001", "120700160010.001", "120801140010.001", "120900670010.001", "120900680010.001", "121000070010.001", "121100020010.001", "121100030010.001", "121100360010.001", "121100370010.001", "121100380010.001", "121100720010.001", "121100730010.001", "121100740010.001", "121100750010.001", "121101080010.001", "121101090010.001", "121200130010.001", "121200140010.001", "121200150010.001", "121200160010.001", "121200170010.001", "121200180010.001", "121300060010.001", "121300070010.001", "121300080010.001", "121300090010.001", "121300100010.001", "121300110010.001", "121300160010.001", "121300170010.001", "121300400010.001", "121300410010.001", "121300450010.001", "121300460010.001", "121300470010.001", "121300480010.001", "121300500010.001", "121300510010.001", "121300560010.001", "121300570010.001", "121300590010.001", "121300600010.001", "121300610010.001", "121300660010.001", "121300670010.001", "121300680010.001", "121400250010.001", "121400260010.001", "121400300010.001", "121400310010.001", "121400320010.001", "121400330010.001", "121400350010.001", "121400360010.001", "121500090010.001", "121500100010.001", "121500110010.001", "121500120010.001", "121500130010.001", "121500190010.001", "121500200010.001", "121500410010.001", "121500420010.001", "121500480010.001", "121500490010.001", "121500500010.001", "121500510010.001", "121500520010.001", "121500530010.001", "121500590010.001", "121500600010.001", "121500610010.001", "121500620010.001", "121500630010.001", "121500630020.001", "121600090010.001", "121600100010.001", "121600110010.001", "121600330010.001", "121600390010.001", "121600400010.001", "121600410010.001", "121600420010.001", "121600430010.001", "121600440010.001", "121600490010.001", "121600500010.001", "121600510010.001", "121600520010.001", "121700040010.001", "121700050010.001", "121700110010.001", "121700330010.001", "121700340010.001", "121700400010.001", "121700410010.001", "121700420010.001", "121700430010.001", "121700440010.001", "121700450010.001", "121700510010.001", "121700520010.001", "121700530010.001", "121700540010.001", "121700550010.001", "121700560010.001", "121800220010.001", "121800230010.001", "121800240010.001", "121800250010.001", "121800260010.001", "121800270010.001", "121900070010.001", "121900080010.001", "122200090010.001", "122200150010.001", "122200160010.001", "122200500010.001", "122200510010.001", "122200520010.001", "122200870010.001", "122200940010.001", "122201030010.001", "122300110010.001", "122300120010.001", "122300460010.001", "122300640010.001", "122300650010.001", "122300660010.001", "122300670010.001", "122400200010.001", "122400210010.001", "122400220010.001", "122400470010.001", "122400480010.001", "122400490010.001", "122400510010.001", "122400520010.001", "122400690010.001", "122401000010.001", "122401010010.001", "122401020010.001", "122401030010.001", "122401040010.001", "122401050010.001", "122500070010.001", "122500120010.001", "122500130010.001", "122500140010.001", "122500150010.001", "122500160010.001", "122500170010.001", "122600120010.001", "122600130010.001", "126100940010.001", "126100950010.001", "126101010010.001", "126101020010.001", "126101030010.001", "126101040010.001", "126101050010.001", "126101060010.001", "126300520010.001", "126300580010.001", "126300590010.001", "126300600010.001", "126300610010.001", "126300620010.001", "126300630010.001", "126300680010.001", "126300690010.001", "126300700010.001", "126300710010.001", "126300720010.001", "126300730010.001", "126400300010.001", "126400310010.001", "126400530010.001", "126400540010.001", "126400550010.001", "126400600010.001", "126400620010.001", "126400630010.001", "126400640010.001", "126400650010.001", "126500620010.001", "127400040010.001", "127500170010.001", "127500180010.001", "127500190010.001", "127600280010.001", "127600290010.001", "127600300010.001", "127600630010.001", "127600640010.001", "127600650010.001", "127600930010.001", "127601020010.001", "127700040010.001", "127700050010.001", "127900520010.001", "127900530010.001", "127900680010.001", "127900770010.001", "128000180010.001", "128000190010.001", "128000200010.001", "128000530010.001", "128000540010.001", "128000550010.001", "128000950010.001", "128001080010.001", "132600640010.001", "132600650010.001", "132600890010.001", "132600900010.001", "133000170010.001", "133000210010.001", "133000220010.001", "133000230010.001", "133000250010.001", "133000260010.001", "133100130010.001", "133100140010.001", "133100150010.001", "133100160010.001", "133100180010.001", "133100190010.001", "133200210010.001", "133200220010.001", "133200230010.001", "133200250010.001", "133200260010.001", "133300200010.001", "133300210010.001", "133300220010.001", "133400040010.001", "133400050010.001", "133400370010.001", "133400380010.001", "133400390010.001", "133400400010.001", "133400720010.001", "133400730010.001", "133400820010.001", "133400830010.001", "133400840010.001", "133400880010.001", "133500150010.001", "133500160010.001", "133500170010.001", "133500180010.001", "133500250010.001", "133500260010.001", "133500540010.001", "133500550010.001", "133500560010.001", "133500890010.001", "133500900010.001", "133500910010.001", "133600280010.001", "133600290010.001", "133600300010.001", "133600630010.001", "133600640010.001", "133600650010.001", "133600870010.001", "133600880010.001", "133600930010.001", "133600940010.001", "133600950010.001", "133600960010.001", "133601030010.001", "133601040010.001", "133700130010.001", "133700140010.001", "133700150010.001", "133700160010.001", "133700230010.001", "133700240010.001", "133700340010.001", "133700350010.001", "133700670010.001", "133700680010.001", "133700690010.001", "133700700010.001", "133701020010.001", "133701030010.001", "133900090010.001", "133900100010.001", "134000200010.001", "134000210010.001", "134000220010.001", "134000450010.001", "134000590010.001", "134000600010.001", "134000610010.001", "134000620010.001", "134000690010.001", "134000700010.001", "134000860010.001", "134000870010.001", "134000880010.001", "134000950010.001", "134000960010.001", "134100200010.001", "134100210010.001", "134100220010.001", "134100510010.001", "134100520010.001", "134100530010.001", "134100540010.001", "134100610010.001", "134100620010.001", "134100790010.001", "134100850010.001", "134100860010.001", "134100870010.001", "134100880010.001", "134100950010.001", "134100960010.001", "134300210010.001", "134300220010.001", "134300230010.001", "134300240010.001", "134300310010.001", "134300320010.001", "134300430010.001", "134300440010.001", "134300760010.001", "134300770010.001", "134300780010.001", "134300790010.001", "134301110010.001", "134400210010.001", "134400220010.001", "134400230010.001", "134400240010.001", "134400310010.001", "134400320010.001", ], "src_name": "Crab", "token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJDYXJsby5GZXJyaWdub0B1bmlnZS5jaCIsImVtYWlsIjoiQ2FybG8uRmVycmlnbm9AdW5pZ2UuY2giLCJuYW1lIjoiY2ZlcnJpZ25vIiwicm9sZXMiOiJhdXRoZW50aWNhdGVkIHVzZXIsIGFkbWluaXN0cmF0b3IsIGNvbnRlbnQgbWFuYWdlciwgZ2VuZXJhbCwgaW50ZWdyYWwtcHJpdmF0ZS1xbGEsIG1hZ2ljLCB1bmlnZS1ocGMtZnVsbCwgcHVibGljLXBvb2wtaHBjLCBhbnRhcmVzLCBzZHNzIiwiZXhwIjoxNjI1MzA2NzYyfQ.pa6HOaVvBOFKdvgilajGsfzDxlGrRsD8bp_jW9AZpCI" , }

data_collection = disp.get_product(**par_dict)

[volodymyrss]

API code will contain an outdated token in several cases.

Of course. The message should be informative, kind of. What was the message?

Even after getting an updated token, I get this error ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Immediately or after a delay?

However, if the URL does not work, I would avoid putting it in the email

it's a bit unclear when it will work, it depends on proxy. Once final url is verified, we make a simple constrain based on that. As I suggested above.

[ferrigno]

API code will contain an outdated token in several cases.

Of course. The message should be informative, kind of. What was the message?

I realized afterwards that the token should have been still valid (48 hours), the error message was the one below.

Even after getting an updated token, I get this error ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Immediately or after a delay?

After a delay, say a minute, but I did not measure.

[volodymyrss]

Even after getting an updated token, I get this error ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Immediately or after a delay?

After a delay, say a minute, but I did not measure.

a minute is the limit at server. It may be exceeded for larger products in some links and proxy configurations. We can increase the timeout.

[ferrigno]

The web interface never gave any result

[ferrigno]

For the API, the time from request to error report is exactly 1 minute import datetime now = datetime.datetime.now() try: data_collection = disp.get_product(**par_dict) final_time= datetime.datetime.now() except: print("Failed") final_time= datetime.datetime.now()

Time of request 2021-07-02 12:58:16.155738 time of failure 2021-07-02 12:59:16.924571 Time difference: 0:01:00.768833

[volodymyrss]

the limit is now 5min and the request passes. Also frontend request should pass if repeated. Request by URL will not work

[volodymyrss]

maybe @burnout87 have a look for first implementation. Else let it wait till I am back on Thursday.

[burnout87]

However, if the URL does not work, I would avoid putting it in the email

So, if I correctly understood, the case of not-working URL happens when the scw list is too long, is it correct ?

If yes, is there a limit for the list that should be take into account?

[ferrigno]

I kind of confirm your last statement, but as far as I understood, this limit is not clear as it depends also on Proxy

[volodymyrss]

However, if the URL does not work, I would avoid putting it in the email

So, if I correctly understood, the case of not-working URL happens when the scw list is too long, is it correct ?

It happens when scw list is passed as a parameter and is too long. But it may also happen in other cases.

If yes, is there a limit for the list that should be take into account?

Limit should be on the URL. The exact limit depends on many factors, but 2000 characters would work everywhere.

This problem here occurs because redirect with resolve-url (which shortens URLs) is rejected, and there should be probably exception in this functionality.

[burnout87]

This problem here occurs because redirect with resolve-url (which shortens URLs) is rejected, and there should be probably exception in this functionality.

So, do we also want this link to be in the email? As @ferrigno was suggesting, if it does not work, then it should not be placed within the email.

[volodymyrss]

This problem here occurs because redirect with resolve-url (which shortens URLs) is rejected, and there should be probably exception in this functionality.

So, do we also want this link to be in the email? As @ferrigno was suggesting, if it does not work, then it should not be placed within the email.

That's what I meant: there should be an exception when leading to special treatment and a different note in the email.