gyang128
commented
1 day ago While this is not a technical issue from our side, it is worthy noting that in our final weeks of data collection, our system was constantly under heavy load (near 100% resource usage 24/7). Despite rebooting the system, we were unable to figure out why the system was under so much strain. While this did raise some issues for us (i.e. scripts not functioning 100% of the time as a result of lack of computing resources), we believe that our data collection process was not severely impacted by it as we were still able to collect tens of thousands of viable data points. We did later identify an issue within our recycling script. The fix we implemented did increase the success rate of our scripts by a bit, but not to 100%.
In our final weeks of data collection, we made little to no changes. Our system architecture and data were not affected by any of the changes. Any changes to our scripts were fixes to pre-existing issues that we have noted in prior tech logs.