search

gmantele / vollt

Java libraries implementing the IVOA protocol: ADQL, UWS and TAP
http://cdsportal.u-strasbg.fr/taptuto/
30 stars 29 forks source link

executionDuration not respected for certain output formats #74

Open almicol opened 6 years ago

almicol commented 6 years ago

Hi Grégory, Depending on the output format selected, a job could still run after having reached the executionDuration limit. This can cause stress to the database server, hence we are going to disable the following formats until a fix is implemented: csv, tsv, text, html, json

Formats which do respect the executionDuration limits are: votable, votable/b, fits

PS: Other formats were not tested.

gmantele commented 6 years ago

I never observed such behaviour before (except maybe with text...but this one is kind of special).

  1. May I ask which version of the library you use?
  2. How much time between the job cancellation and its effective end?
  3. How did you test that? Could I reproduce this procedure easily? It will be really useful to detect the origin of the problem.
vforchi commented 6 years ago
  1. we are using my fork on github
  2. I think the job never ends
  3. It is always reproducible, you just need a long running query (or to decrease the execution duration, but we always tried with the default, 60s)
gmantele commented 6 years ago

I did not succeed to reproduce the same problem as you.

Could you try with a more recent version of the library (i.e. v2.1 or v2.2)?

I indeed already fixed few things relative to the abortion of jobs (see News about v2.1, especially "Fix abortion of a query" and "Fix DB transaction end")

gmantele commented 6 years ago

Are you still observing this problem?

vforchi commented 6 years ago

Yes, but I haven't upgraded from the master.

gmantele commented 6 years ago

I just tried again, but I was not able to reproduce this issue.

I tried with CSV, first with a default timeout of 3 seconds, then of 30 seconds. The SQL query execution was very fast, but not the generation of the result. The job was canceled during the result formating, as, I assume, it should be in your case.

gmantele commented 6 years ago

I don't know if you have already tested with a more recent version of the library, but another possibility would be a sync_fetch_size and/or async_fetch_size too large. Generally, a good value is about 100 or 1000.

What value do you have for these properties?

gmantele commented 4 years ago

@vforchi and @almicol , do you still observe this problem? Even after an update of TAPLib?

If no answer in some days or if the answer is "no", I will close this GitHub Issue.

almicol commented 4 years ago

Sorry Gregory both Vincenzo and I are on holidays. Please do not expect an answer before the end of next week. Thanks, Alberto

On 21 Aug 2020, at 11:37, Grégory Mantelet notifications@github.com wrote:

 @vforchi and @almicol , do you still observe this problem? Even after an update of TAPLib?

If no answer in some days or if the answer is "no", I will close this GitHub Issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.