dolthub / dolthub-issues

Issues for dolthub.com
https://dolthub.com
4 stars 1 forks source link

remove query row limit for csv download #495

Closed g-sando closed 9 months ago

g-sando commented 1 year ago

Currently only 200 rows are returned for any executed query other than the default table queries, which means only 200 rows can be exported to csv without using dolt in the command line. This export of a View/Saved Query or API call is the main and most basic functionality that allows users in our org who aren't devs to access our data autonomously.

Request: Remove the query output limit to allow full query and API call downloads to csv.

Describe alternatives you've considered Using the CLI. The need for this makes the end-to-end user-friendliness for users who aren't familiar with the CLI very low.

image

timsehn commented 1 year ago

As a workaround, you can have the user download the full CSV for the table and perform whatever filters they need on the full table.

The problem on our side is that the query can take more time than a reasonable webpage load if we remove the row limits. We can't stream the results like a full table.

Good feature request though. I definitely wish there was a better way here as well.

liuliu-dev commented 1 year ago

Hi @g-sando, thanks for the feature request. The nearest at hand solution for this would be increase the row limit, potentially setting it to 1000, would that be enough for you? Or we can create a job and support unlimited downloads, however, this will take more time to implement. Which approach suits better with your needs?

g-sando commented 1 year ago

Hi @liuliu-dev sorry for the delay, increasing to 1000 for now would still be great and still increase our flexibility by a lot, thanks so much.

Unlimited would be ideal long-term but I understand the constraints Tim mentioned so can circle back if that becomes unavoidable down the line!

liuliu-dev commented 1 year ago

@g-sando I've increased the row limit to 1000. If you have any further questions or requests, feel free to reach out. Thanks for your understanding!

Screenshot 2023-09-07 at 10 49 00 AM

liuliu-dev commented 9 months ago

Hi @g-sando , now you can create a query job and have unlimited downloads! You can find the 'Create Query Job' button in the SQL editor here: Screenshot 2023-12-11 at 12 52 51 PM

Once the job is done, you'll find a download link for the result on the job page: Screenshot 2023-12-11 at 12 56 52 PM

I will close this issue as completed :)