sleeping-koala
commented
3 years ago Onboarding Documentation: So far, I've completed the framework for the python script, which I stopped maintaining since our client was dropped. The script is located under my personal repo at https://github.com/jsflca2012/spreadsheetScripts. Feel free to clone this repo and make any modifications as you see fit. The script essentially uses the openpyxl library to scrape through each column and run test validation methods on each column. The test methods are not completely built out, so you'd have to consult with the design and backend team to ensure that the information conforms with the desired format. Currently, if the script finds any errors it will store it in an array of arrays containing the row number with the error along with what went wrong. The way I have it working right now is very primitive and can be rewritten in order to cover a wider breadth of cases. The script as it currently stands should work for in-house usage, but making it more modular in order to easily adapt to format changes should be taken into consideration to minimize tech debt and allow for easier future changes.
The format for the data we discussed is located under the drive folder https://drive.google.com/drive/folders/1Tc45Tl6LWRbnyGfafusJGmI_32WObsT5?usp=sharing. If you have any questions, reach out to Ryan and Katie to discuss formats appropriate to the handled data.
I never implemented this, but I remember discussing that the script should also record what % of files fail in order to prevent automatically sending tons of bad data to the backend and breaking the app. I imagine this could be easily done using the array of failures I built out and dividing it over total row entries. The exact % or number of failures that prevents the data from being sent is still undecided, consult with the design and backend team to determine how to handle this exactly.
Additionally, backend implementation is not developed for this script yet, so you would have to figure out an efficient way to connect the backend with this script's output. Keep in mind you should not be sending the data immediately, since you would want to run the script in order to check for errors then determine whether the data is reliable enough to be passed to the database. To be safe, I would separate out the upload function into a different method, or perhaps a different script altogether.
From what I remember, the frontend interface seems to be mostly built out. However, the frontend is currently unlinked with the backend, so I'm not exactly sure how the data from the frontend will be transferred over to this script. You can consult Ryan about this, who built out the frontend interface, in order to determine the best course of action for this.
Description
To finish the flow of importing existing business data to Omou
Frontend Dev Todos
Backend Dev Todos