IATI / D-Portal

http://d-portal.org/
Other
30 stars 23 forks source link

D-query hangs when running example code #653

Open robredpath opened 1 year ago

robredpath commented 1 year ago

Brief Description

Using the menu controls to populate the query box with the "select organisation" example query and then attempting to run it results in the application hanging

Severity Medium

Issue Location http://d-portal.org/dquery/

Steps to Reproduce

Go to http://d-portal.org/dquery/ Click Examples -> Select Organisation Click Run

Expected Results/Behaviour Query results to be displayed in the right-hand pane

Actual Results/Behaviour

Screenshot 2023-11-28 at 09 30 55

Notes

This happens in both Safari and Chrome on macOS (latest everything, running on Intel). No adblocking.

Other queries work fine.

xriss commented 1 year ago

Not sure how to fix, this gets 10 random org files, however some org files are huge and if we are unlucky we get some of the big ones as results and then the browser runs out of memory displaying it.

The unluckiness happens at import time not at query time so it is currently locked in until tomorrow.

Maybe including an "order by pid" to stop it being random?

https://d-portal.org/dquery/#%0Aselect%0A%0A*%0A%0Afrom%20xson%20where%20root='/iati-organisations/iati-organisation'%0A%0Aorder%20by%20pid%20limit%2010;%0A%0A

xriss commented 1 year ago

Since this is a data problem other possible solutions are.

  1. Use a simple text only view if the result of a query is bigger than X where X is a large number. Should work, might be annoying if we pick too small a large number.

  2. Switch to a spread sheet view, which I would like to do but have not found an open source javascript in browser spreadsheet library that I like, and of course this may still go horribly wrong with large query results depending on the library.

  3. Tell publishers not to make such huge org files and give them a way to report organisation level budget data inside multiple files.

I advise we do all 3 :)

robredpath commented 1 year ago

Thanks, @xriss !

If I click the link above then click "Run" I get a very similar problem: it appears to hang (long enough that I gave up and started writing this update...), but it does eventually complete. Maybe I'm only moderately unlucky today?

I like those solutions, @xriss - a simple text view for large results is fine, IMO, as would not trying to render the results at all but letting the user download them as a text file (with an appropriate message in the right-hand pane).

As a further alternative, we could change the example query to be what I was expecting when I clicked the button, which was a query of the form "SELECT * FROM organisations WHERE org-id = "GB-COH-12345" . Presumably we can then choose an organisation that we know has a sensible query size.

notshi commented 10 months ago

Thanks, @robredpath - the example query is now updated with your suggestion.

I've also added comments for all the example queries.