When the 'Run scrape' button is pressed, it spins for several minutes before the 'All records scraped successfully' message pops up.
The backend logs Upserting {number_of_reports} reports and Upserting report {index} of {number_of_reports} but these aren't visible to the frontend.
These updates would be useful if displayed on the frontend, to show the user how much of the scrape has completed. It also gives evidence that the scrape is still running and hasn't got stuck.
Ideally we wouldn't be polling the backend continuously; a possible way to get this working would be using websockets.
When the 'Run scrape' button is pressed, it spins for several minutes before the 'All records scraped successfully' message pops up. The backend logs
Upserting {number_of_reports} reportsandUpserting report {index} of {number_of_reports}but these aren't visible to the frontend. These updates would be useful if displayed on the frontend, to show the user how much of the scrape has completed. It also gives evidence that the scrape is still running and hasn't got stuck. Ideally we wouldn't be polling the backend continuously; a possible way to get this working would be using websockets.