Write progress to a DB table

carlosms commented 5 years ago

The command will write in a table the number of total/done resources.

smacker commented 5 years ago

JFYI, gitcollectors saves progress like this:

se7entyse7en commented 5 years ago

My initial idea was to have something like this:

id     | done | total
=====================
global |    1 |     3     => dashboard: 33.3%
src-d  |  100 |   100     => dashboard: 100%
bblfsh |    3 |    10     => dashboard: 30%
google |    0 |  null     => dashboard: 0%

But now I realzied that the first special row is actuall non-necessary as it can be inferred by the other rows, and we can just have:

org     | done | total
=====================
src-d  |  100 |   100     => dashboard: 100%
bblfsh |    3 |    10     => dashboard: 30%
google |    0 |  null     => dashboard: 0%

which is also more similar to the one you posted @smacker.

carlosms commented 5 years ago

We may also want to add the failed column, as already discussed in #37.

To simplify code and work incrementally, maybe for this PR we don't change the failure handling. Right now we exit on the first failure found, and it's OK if the code writes this 1 failure on the DB and exits.

Then later we can think of a PR where we may want to log the failures and continue with the rest of the resources.

src-d / ghsync

Write progress to a DB table #31