search

participedia / usersnaps

Where usersnap files bug reports
0 stars 0 forks source link

[USERSNAP] Application error when clicking the results CSV for cases. Download does not work. #1888

Closed paninee closed 1 year ago

paninee commented 1 year ago

Sender: jesi.carson@gmail.com Taken at: https://participedia.net/search?selectedCategory=case Fields:

SCREENSHOT Download original image

Browser: Chrome 109 (macOS Catalina) Screen size: 1920x1080 Browser size: 1375x825 Open #1571 in Usersnap

Powered by Usersnap.

jesicarson commented 1 year ago

screenshot of the error:

Screen Shot 2023-01-24 at 11 51 50 AM
jesicarson commented 1 year ago

@paninee do you have a recent export i can email to the user?

jesicarson commented 1 year ago

I found one from April 2022 and emailed it to them for now. but it would be good to have the most up to date one.

jesicarson commented 1 year ago

Another user emailed about this today. @paninee Can we move this up the priority list after adding captcha to preview? Thanks!

paninee commented 1 year ago

@jesicarson This is caused because we export too many records. We don't have enough memory to do that. We have 2 choices to solve this:

  1. Increase our server's capability. Which increases the monthly cost, probably by $25/month. While it's the quickest fix, this might not be a permanent solution as our data keeps growing.
  2. Force the user to apply filters, and set a limit of how many records they can download. This may not work for some researchers. Took time to implement, probably about 20-30h.
  3. Make the CSV download a background task. We work on packaging up the CSV in the background (maybe use worker dyno in Heroku in the future to prevent the main web dyno from being too busy). Store it in S3. Then the user can check if the CSV is completed and ready to be downloaded. similar to how we backup our database. It takes a long time so Heroku shows the progress, when it's done, the status got updated to "Ready". Then we can download it. More complex feature. Probably takes about 60-80h of work.
jesicarson commented 1 year ago

Paul agrees with the decision to pay to upgrade the DB for a month or two, and see how long we can get away with that.

Next step would be to scale back the feature - only allow users to download the dataset once they've applied a certain number of filters / limit the results + provide the full dataset csv for cases manually every month or so as a hyperlink on the research page.

(note that methods and orgs probably don't have this issue)

jesicarson commented 1 year ago

@paninee please let me know when the DB has been updated so i can test if it works

jesicarson commented 1 year ago

@paninee did upgrading the database work? we should chat about this, especially the cost implications. ill put this on the weekly agenda for monday.