API upload to entries endpoint get rejected

[ondrej1024]

Describe the bug I am trying to upload an SVG data record to the entries API but it gets rejected. The API response is the data record I am trying to upload. But there is no message indicating why the record gets rejected. So there must be something wrong with the request and I would like to know what it is.

To Reproduce Steps to reproduce the behavior:

I send the following API request to my Nightscout instance at 10.5.0.12:8080

curl -X POST -H 'api-secret: cc8a9a7c3670b6d3966fff040238f01a99da6b38' -H 'accept:  */*' -H 'Content-Type: application/json' -i 'http://10.5.0.12:8080/api/v1/entries.json' --data '[{\"type\": \"sgv\", \"date\": 1573564560, \"sgv\": 111}]'

The response is the following:

HTTP/1.1 200 OK
X-Powered-By: Express
Last-Modified: Mon, 19 Jan 1970 05:06:04 GMT
Vary: Accept, Accept-Encoding
Content-Type: application/json; charset=utf-8
Content-Length: 96
ETag: W/"60-7sb1DtVf29uxN9fI8jx3sIpaEgw"
Date: Tue, 12 Nov 2019 13:20:42 GMT
Connection: keep-alive

[{"type":"sgv","date":1573564560,"sgv":111,"utcOffset":60,"sysTime":"2019-11-12T13:20:42.600Z"}]

Expected behavior I expect an empty response which means the data record was accepted.

Your setup information

• Version of Nightscout: 0.12.5
• Environment:

export DISPLAY_UNITS="mg/dl"
export MONGO_CONNECTION="mongodb://user:xxxx@localhost:27017/Nightscout"
export MONGO_COLLECTION="entries"
export BASE_URL="http://my.nightscout.com"
export PORT=8080
export API_SECRET="1234567890AB"
export INSECURE_USE_HTTP=true
export AUTH_DEFAULT_ROLES="readable devicestatus-upload"
export PUMP_FIELDS="reservoir battery status"
export ENABLE="careportal basal rawbg cob iob cage bwp upbat sage pump devicestatus"
export DEVICESTATUS_ADVANCED=true
export TIME_FORMAT=24

[sulkaharo]

200 OK indicates the insert succeeded. The API responds with the entries that were created.

[ondrej1024]

The API documentation says the contrary:

Also I don't get any SVG value on the Nightscout WebGUI So it looks like nothing gets stored in the DB.

[steve8x8]

In my copy of the API docs (via Swagger), it's api_secret not api-secret, so there must be a mistake on one of both sides...

[steve8x8]

Also, date seems to be in milliseconds (here the docs seem to be wrong, claiming that the value was an "Epoch" one - which is full seconds after Jan 1, 1970).

[ondrej1024]

Yeah, unfortunately the documentation is the week spot of Nightscout :cry: The Swagger doc is wrong and I found the correct syntax here by chance:

So the problem I raised here is not an authorization issue because this would cause a 401 (Unauthorized) HTTP response.

So I wonder if there are other errors in the api docs. For example non of the parameters in the entries upload request is marked as required but the api response with an error if the date is not provided. In my case I an sending the date but maybe my request is missing some other parameter which causes it to be rejected ?

Can anyone from the developers please have a look at this and give me a clue why my request gets rejected?

[sulkaharo]

PR to fix the swagger file is welcome!

[ondrej1024]

Also, date seems to be in milliseconds (here the docs seem to be wrong, claiming that the value was an "Epoch" one - which is full seconds after Jan 1, 1970).

That was it! :smile: The date parameter needs to be sent as Epoch * 1000. Thanks for that clue. Now it is working.

Finally I get the sgv value I am sending in the GUI. However the data record is still sent back in the API response. So it seems the docs are wrong also in this case.

Looks like a request with wrong date parameter is silently discarded by the server, and the API response is always the same in both success and failure case.

[steve8x8]

Confirmed. While I was experimenting, I always got a 200 response with the (somewhat enhanced) record back. In case of success, a list of _ids might be more useful, btw. Now I've got to clean up multiple test records from the db...

BTW, even a date divided by 1000 is still a vaild date, just a bit closer to the Epoch start. Does a record actually get rejected if date and dateString don't match?

[ondrej1024]

BTW, even a date divided by 1000 is still a vaild date, just a bit closer to the Epoch start.

Actually this is true. The Epoch date of today divided by 1000 would be on 19 Jan 1970. So probably during my test the records didn't get rejected by weren't displayed on the GUI because they were too old. I should check the Mongo DB directly.

Does a record actually get rejected if date and dateString don't match?

What would be the reason to send both? Isn't one of them sufficient? Again, this should be specified in the api doc so less guessing would be necessay.

[steve8x8]

Here's what I found - there seems to be nothing before 2019, this seems to indicate that date values are somewhat checked, but they may actually diverge from what the dateString suggests :(

[{"_id":"5dcd28385ce59abea11079cd","device":"glimp://libre/0X00UNKNOWN","type":"sgv","date":1546340400666,"dateString":"2019-01-01T11:00:00.000Z","sgv":123,"utcOffset":0,"sysTime":"2019-01-01T11:00:00.000Z"},{"_id":"5dcd25f95ce59abea11076fb","device":"ohm meter","type":"mbg","date":1546340400000,"dateString":"2019-01-01T11:00:00.000Z","utcOffset":0,"mbg":123,"sysTime":"2019-01-01T11:00:00.000Z"}]

and

[{"_id":"5dcd2e2d5ce59abea110809c","device":"test","type":"sgv","date":1561975210666,"dateString":"2019-07-01T10:00:01.666Z","sgv":123,"utcOffset":0,"sysTime":"2019-07-01T10:00:01.666Z"},{"_id":"5dcd2bec5ce59abea1107e41","device":"test","type":"sgv","date":1561975200666,"dateString":"2019-07-01T10:00:00.000Z","sgv":123,"utcOffset":0,"sysTime":"2019-07-01T10:00:00.000Z"},{"_id":"5dcd2c7d5ce59abea1107ed2","device":"test","type":"sgv","date":1561975200666,"dateString":"2019-07-01T10:00:00.666Z","sgv":123,"utcOffset":0,"sysTime":"2019-07-01T10:00:00.666Z"},{"_id":"5dcd2f575ce59abea11081ef","device":"test","type":"sgv","date":0,"dateString":"2019-07-01T10:00:02.666Z","sgv":123,"utcOffset":0,"sysTime":"2019-07-01T10:00:02.666Z"}]

it's not clear to me whether the device is actually required, and I won't add more junk to my data for now... until I find (or get told, hint hint) how to clean up that mess.

[sulkaharo]

Ah, by default the /entries API limits the data being loaded if no date query is being sent. If you want to load old data, you'll have to issue a date query, for example /api/v1/entries/sgv.json?count=1000&find[date][$lt]=1550575829727

[steve8x8]

@sulkaharo this is a rather surprising, and certainly undocumented, behaviour. Violation of POLA, in other words. Please get it documented (by someone who knows the inner workings by heart, not some innocent user). Mess cleaned up. I presume there's no protection against duplicate entries, btw?

[ondrej1024]

I think we can close this issue as it is clear now that it is not a bug in the Nightscout server code but rather lack of documentation.

Unfortunately the API documentation is both incomplete and erroneous. This makes it very difficult for third party clients to use it unless you have the knowledge and the time to dig into the implementation details of the server code itself.

[steve8x8]

Just to settle the discussion whether dates not multiplied with 1000 would show up in the db or not - I found this while searching for magic numbers:

./lib/api3/const.json:  "MIN_TIMESTAMP": 946684800000,

946684800 is the number of seconds between the epoch and Jan 1st 2000 UTC. This means, dates in the previous millenium would not be accepted (at least unless there's a valid dateString? I was able to insert a record with "date":0).

[ondrej1024]

Created new issue #5221 regarding the documentation.