We want to make it easy for anyone to transfer or upload files to estuary and one of features we can introduce for this is to allow uploading files with URLs.
The idea is to allow the user to specify a link (URL), which the process can grab and upload it to estuary using the API endpoints.
Changes
add a downloader queue that will consume these URLs.
the queue will download the file (wget or curl) and load it on estuary using the endpoints.
make a change on the UI to allow uploading files.
Notes
make sure to check for duplicate files
the downloader can be another process or module that sits alongside estuary.
each download should call a goroutine and can emit responses (use socket.io)
Proposal
We want to make it easy for anyone to transfer or upload files to estuary and one of features we can introduce for this is to allow uploading files with URLs.
The idea is to allow the user to specify a link (URL), which the process can grab and upload it to estuary using the API endpoints.
Changes
Notes