Automating backend data pipeline

[JoannaBlatt]

This includes:

• automating input pipeline
• automating output handling
• automating file deletion

[JoannaBlatt]

Automated pipeline stopped working. I think because it was moved into the backend folder, but I'll try and troubleshoot it.

[JoannaBlatt]

fixed the issue. I think while I was moving files it refactored the utils import in other files (I must have moved it first) and then didn't un-refactor when everything was in the same place. Back to testing current automation then file removal.

[JoannaBlatt]

Works except ran into an issue because the pareto front drawings end up in their own directory. This causes an issue because I need to automate accessing that directory. Will look at what is used when creating the directory to see if I can reverse the process. Will google first to see if there is an easy solution. Although once we have a way to create session identification that should make it easy since you would just delete anything tagged with that. Could also use the automated directory creation to create directories for each person. Might also be worth changing so that it doesn't have as many sub directories since we aren't using them, but maybe leave in for now in case we use the unprocessed large file and need everything.

[JoannaBlatt]

Data pipeline for single arbor file has been fully automated. It is completely scripted at this point and so I will be closing the issue. If making it user specific needs to be implemented a new issue will be created.

[JoannaBlatt]

https://www.tutorialspoint.com/How-to-delete-all-files-in-a-directory-with-Python Resource that was used.