[User Dashboard] Add Export Function for Past Searches

[aliayar]

As we currently don't have a way for our users to access to their past searches, one of our users asked for implementation of exporting past searches in CSV format.

Edit: Intercom Thread

[schaferyan]

Related to #65 and #685

[hartator]

As we currently don't have a way for our users to access to their past searches, one of our users asked for implementation of exporting past searches in CSV format.

Can you share the Intercom thread?

Related to #65 and #685

👍

[aliayar]

Can you share the Intercom thread?

I have added it to the original post.

[schaferyan]

Another customer asked for a way to download all past searches:

Intercom

[martin-serpapi]

Another customer requested this:

Intercom

[hilmanski]

A user requested to have a bulk export for searches.

Intercom

[schaferyan]

Another customer requested this:

Intercom

[djsegal]

To address the bulk data export need, I suggest creating a user-specific subdirectory in S3, identified by their UUID. Users would have the ability to access and download their data, conveniently zipped, via their API token. This method not only secures data access on an individual level but also streamlines our data management process.

This approach caters to the demand for an efficient export mechanism while upholding user privacy and security. Considering our 30-day data retention policy, implementing this solution can commence promptly without the concern of handling older data

[ilyazub]

@djsegal can you please check if S3 can gzip that user subdirectories with files on AWS side?

[djsegal]

@ilyazub, AWS S3 can handle gzip compression of user subdirectories via S3 Object Lambda. This service enables us to apply custom code to process data during S3 GET, LIST, and HEAD requests, including gzip compression. For a detailed overview, the AWS documentation provides comprehensive insights: Transforming objects with S3 Object Lambda.

Setting this up involves configuring a Lambda function for the specific task of compression. I'm ready to assist in integrating this feature with our system if needed. It aligns well with our focus on efficient data management and could streamline our operations significantly

[ilyazub]

AWS S3 can handle gzip compression of user subdirectories via S3 Object Lambda.

After the discussion with @hartator and @djsegal, we found out that S3 Object Lambda won't work for async=true searches. async=true searches are parsed on-demand when the JSON endpoint is accessed.

---

Cursor pagination would work better for us (ref: https://github.com/serpapi/SerpApi/pull/2114#issuecomment-1032249855).

Example possible request:

GET https://serpapi.com/searches?starting_after=65faef94260ca965fabad287
# or
GET https://serpapi.com/searches.json?starting_before=65faef94260ca965fabad287

Adding where(:id.lt => params[:starting_after]) or where(:id.gt => params[:starting_before]) selectors in SearchesController#index is enough to implement such a pagination.

MongoDB processes such query within 500 ms even when returning 1,000 searches. (Performance may be improved even more because MongoDB examined 25 * limit keys/documents.)

[hartator]

Cursor pagination would work better for us (ref: serpapi/SerpApi#2114 (comment)).

Example possible request:

GET https://serpapi.com/searches?starting_after=65faef94260ca965fabad287
# or
GET https://serpapi.com/searches.json?starting_before=65faef94260ca965fabad287

Yes, let's do that first. Just returning JSON search list with pagination through MongoDB search IDs and we also want to add its documentation.

Users can then retrieve each search with the retrieved search IDs. Zipping our side will be complicated and limited anyway.

[martin-serpapi]

Another customer requested this:

Intercom

[schaferyan]

Another use case for this:

Intercom

[hilmanski]

Another customer ask for this:

Intercom