Response Object

[Gyukat]

I think it is time to create the "excel/csv" type response object. This would have all data submitted by the agency in one session returned as a spreadsheet document.

It will also include:

• validation success/failure > reason for failure
• if successful:
  • report number
  • any created DEQARINSTID
  • flags > reason for flag
• (Other?)

In the email body we can put the overall counts...and possibly other statistics we can think of.

A session can be defined by us for each type of upload:

• Manual (per logged in session OR per day?)
• CSV (per sheet)
• JSON (per hour or day?)

[Gyukat]

We may consider returning the JSON object instead of a CSV in response to the API submission.

[Gyukat]

I think the whole question of serial submission could be looked at. We had some weird problems because records go one after the other. Like flags on only one record, or data overwriting data in the same set, etc.

[JoshBone]

I think the whole question of serial submission could be looked at. We had some weird problems because records go one after the other. Like flags on only one record, or data overwriting data in the same set, etc.

Hmm... Would be good if we can find particular examples (at least report id-s). We have a full submission package log, so I could check if it's a package error or some logical error on the backend.

[Gyukat]

I think these are no errors...these were exactly the problem of putting the batches as consecutive files. So, for instance, instead of noting that there were two files with the same ID, it would just overwrite one. There were several problems related to consecutive processing; I will try to remember more.

[JoshBone]

Hmm... Is it API submission or mostly CSV? What if we would extend our CSV submission in a way, that there would a separate menu for uploading a CSV file (with a test first option). And there would be a new menu for batch update which could work in a way, that in step one you are downloading an already pre-filled csv with all the data and you can update and re-upload that. (We can even think about a built in spreadsheet to do the batch updates...)

[Gyukat]

That's cool but I think it is not the exact problem with batch upload...probably most will just want to overwrite existing data, no?

The problem we noted was related to the serial nature of CSV upload---let's focus on the response object first and keep in mind other problems in the longer term.