fecgov / fecfile-web-api

Back-end API for FECfile application
Other
8 stars 2 forks source link

SPIKE - Develop a technical strategy for uploading and tracking pdf files #992

Open exalate-issue-sync[bot] opened 1 month ago

exalate-issue-sync[bot] commented 1 month ago

The FECFile 8.5 spec allows for users to upload a PDF to a report and submit it to EFO.

This ticket is to look at the Django options and Cloud.gov infrastructure for uploading and managing files in Cloud.gov and the process by which they will be added to the payload sent to EFO upon submitting a report.

QA Notes

A write up in the comments describing how the PDF upload will be managed will pass this SPIKE ticket.

DEV Notes

We already manage *.fec files using Postgres and S3 buckets. The solution should leverage that code and add in the best practices for Django file uploading.

Where are we going to store the files?

How are we going to control access?

How will the user upload the file (identify a front-end feature)?

Design

null

See full ticket and images here: FECFILE-1503

exalate-issue-sync[bot] commented 1 month ago

I researched file management in django and found it to be fairly straightforward. More details get be read here: https://docs.djangoproject.com/en/4.2/topics/files/#file-storage

I have a branch created that allows uploading files from the frontend to the back. And set it up so that PDFs are a table in the database associated in a many-to-one relationship with reports, meaning a report could have many pdfs associated with it. The pdfs are stored on the server in the media directory, which can be customized. However, as we already have S3 configured, we can instead set this up to upload the files to our S3 bucket through adding DEFAULT_FILE_STORAGE = "storages.backends.s3boto3.S3Boto3Storage" to our django settings. I wasn’t able to test this to confirm but from what I’ve read that’s all that should need to be changed. 

I was able to do even less to test out the functionality for being able to send a PDF to EFO as part of the report submission process. However, this should be fairly straightforward, with a caveat that it might require them to update their SOAP client to allow us to send this data over. I wasn’t able to directly access their SOAP client in order to check all the options they provide, but they need to provide us the option to send it over. As long as that’s there, we can easily send over the pdf as bytes similar to how we are sending over the dot fec file. I have code in place that shows how this could be done.  I’ve opened up a couple branches that includes my changes. This isn’t meant to be used for merging in, but more to provide a review so people can see and review what I’ve done as a potential process we can use once we do decide to go forward with this.

API: https://github.com/fecgov/fecfile-web-api/tree/pdf-test APP: https://github.com/fecgov/fecfile-web-app/tree/pdf-test

exalate-issue-sync[bot] commented 4 weeks ago

We’ll need to look at the SOAP issue mentioned and have a way to address it as part of this ticket. [~accountid:712020:2a1493e5-adee-45bd-b27e-868a5c8d3f62] may have some suggestions about this.

Moving back to In Progress and assigning to Sprint 48