backend overloaded

[markziemann]

The workload on the backend is too much, need to streamline dee_pipeline.R and dee_progress.R to make them faster/better.

[markziemann]

progress script has been made weekly instead of daily.

future work will merge the two scripts into one

[markziemann]

Both scripts are merged, but there is still a lot of room for improvement. Here are some suggestions from Antony:

• Make 3 column tables which can be easily converted to database
• Most of the backend effort is spent checking line numbers, this could be cut down substantially if it was only done for new datasets.
• Keeping a table of completed datasets is a good idea (athaliana_validated.txt), but need to ensure that they aren't modified any more (could chown to root or use chattr cmd)
• Once database is set up newly validated datasets can be appended quickly, reducing overhead by about 90%

[markziemann]

The suggestions have been implemented in the previous 2 commits and testing will continue.

[markziemann]

testing has completed, working nicely, and efficient