FLARE-forecast / FLARE-containers

Docker containers to run FLARE forecast
0 stars 2 forks source link

Handling Corrupt Git Objects #41

Open vahid-dan opened 3 years ago

vahid-dan commented 3 years ago

Once in a while, we may get errors stating "corrupt Git objects" in our local Git repositories on the gateways while running the Git script to commit and push data and logs to the remote Git repository at GitHub. For instance:

error: object file .git/objects/ff/970d54651e8410eb3b55fe04b6a1af55dba832 is empty
fatal: loose object ff970d54651e8410eb3b55fe04b6a1af55dba832 (stored in .git/objects/ff/970d54651e8410eb3b55fe04b6a1af55dba832) is corrupt

This error prevents the data from being pushed to the remote repository. It happens much more often on battery-powered gateways compare to AC-powered gateways. We are not sure about the cause of these problems, but our current assumption is that they happen due to an ungraceful exit caused by a system shutdown while the Git process is still running. While it seems we cannot prevent it from happening, we can detect it from the Git logs and resolve it automatically without manual intervention.

Here is the proposed workaround: 1- Check the Git logs from the previous run before running the Git script. 2- If there is a fatal: loose object error at the tail of the log file: 2-1- git clone the branch again with a new temporary name. 2-2- Copy the log files (excluding .git files) from the current Git branch to the new temporary directory. 2-3- Delete the current local Git directory. 2-4- Rename the temporary directory to match the previous local Git directory you just deleted.

There must be other ways to overcome the problem, such as the one suggested here. But, I couldn't find any other fast and straightforward solution.