Implement IPS Grow-like upload system to avoid using the backend

[cyramic]

To make sure upload files aren't limited by the backend, a more direct route may be needed. To do this, we can use a setup similar to what we've done on IPS grow.

1) Create AWS key for uploads, to be added as environment variables. 2) Implement an AWS app like this: https://github.com/SocialFinanceDigitalLabs/ips-reporting-tool/tree/main/aws 3) Generate Pre-signed URLs like this: https://github.com/SocialFinanceDigitalLabs/ips-reporting-tool/blob/main/ips_app/views/upload.py#L70

[cyramic]

Francisco says, "basically you just need to have an API endpoint in the backend that generates the presigned urls. The frontend does all the rest: when the user uploads, it calls that API endpoint with some ID (filename or something) - not with the file itself. The backend generates the presigned urls and returns them. The frontend then uploads to AWS directly, using those urls. The good part of the presigned urls is that they are temporary and will only work for the specified key created (edited) just to keep in mind that each file needs a url - there's no option of having a presigned url for multiple files, I think"

[MagicMiranda]

Copying over code from IPSG Reporting Tool .... seems relatively straightforward. Probably about 1 day. Matthew and Tambe.

Quick fix is config change - changes number on the setting..... but user won't know if they hit the limit. Risk: Users can continue uploading larger and larger files. Cost implication of larger and larger files/ memory etc.

1st: Config limit change later: IPSG code copied over to avoid cyclical heavy load on memory and costs etc.

[franciscobmacedo]

Here's a list of steps that need to be done from IPS:

1. copy the aws app from IPS - this is an independent app, that has the S3 service to create the presigned urls and also some dummy views so this can work with the local filesystem in development and in the tests. Part of this code is probably not required, such as read_file, delete_file because we also have the pyfilesystem2 here, so this can be renamed or reorganised. Check how the dummy or "real" service is used in the settings (development and production)

2. The S3 service can be seen in action in this view. this view is only called with filenames, so the file data never goes through the server. In IPS, we create a model instance for each upload (so it can be processed later), but this doesn't seem to be necessary here. I believe something like this should be enough:

   @require_POST
   def presigned_urls(request):
   body = json.loads(request.body)
   file_names = body.get("files", [])
   s3 = settings.S3_SERVICE()
   files_presigned_urls = []
   for filename in file_names:
       file_key =generate_unique_key(filename)
       url = s3.put_presigned_url(file_key)
       urls.append({'filename': filename, 'url': url})
   return JsonResponse({"urls": urls}, status=200)

3. In the frontend, when the user clicks "upload" we need to call the above mentioned view and, for each url in the response, we upload the corresponding file to AWS directly, like it's done here

[MagicMiranda]

https://socfinuk.atlassian.net/browse/SFDPCM-18

https://socfinuk.atlassian.net/browse/SFDPCM-21