atviriduomenys / spinta

Spinta is a framework to describe, extract and publish data (a DEP Framework).
MIT License
10 stars 4 forks source link

Push status file #510

Open sirex opened 9 months ago

sirex commented 9 months ago

When running spinta push command save push results to a push status file. Later this file can be used to inspect which datasets where pushed successfully, which failed, how many rows were pushed, how many errors, how much time it took to push each dataset.

Status file could look liek this:

{
  "time": "2000-01-01T00:00:00+00:00",  # Time, when status file was updated
  "models": {
    "datasets/gov/example/City": {
      "count": 1000,  # Number of rows to push.
      "pushed": 998,  # Number of rows pushed.
      "errors": 2,    # Number of rows ended with an error.
      "time": 5,      # Time in seconds took to push all rows.
    }
  }
}

This file by default could be stored in ~/.local/share/spinta/status/target.json, where target is target server where data is pushed.