Test GET v1/data

[karimkawambwa]

Description

Ability to retrieve raw data using /v1/data

http://staging.api.sensors.africa/v1/data/

Type of change

• [x] Bug fix (non-breaking change which fixes an issue)

Checklist:

• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation

[kilemensi]

From the wiki:

http://api.airquality.codeforafrica.org/v1/data/ - all measurements for all publicly accessible sensors in our database

If we have 1000 sensors sending data every five minutes, we're talking 1000*24*(60/5) = 288,000 data points per day. Assuming 1 data point is 1KB, we are talking roughly 250 MB of data per day. In a week, that's over a GB, etc. Have you put in place the necessary "DevOps" to handle such loads @karimkawambwa?

[karimkawambwa]

If we have 1000 sensors sending data every five minutes, we're talking 1000*24*(60/5) = 288,000 data points per day. Assuming 1 data point is 1KB, we are talking roughly 250 MB of data per day. In a week, that's over a GB, etc. Have you put in place the necessary "DevOps" to handle such loads

@kilemensi there is a separated load test story on PT and it currently does paging so the data is not returned all at once. Maximum 100 data points for each call. I will address server load on the separate issue.

[kilemensi]

This looks like something we should be implementing in sensors.AFRICA-AQ-api and sending a PR back to feinstaub though i.e. it's a bug in feinstaub itself. Unless we know they don't want it fixed.

[karimkawambwa]

This looks like something we should be implementing in sensors.AFRICA-AQ-api and sending a PR back to feinstaub though i.e. it's a bug in feinstaub itself. Unless we know they don't want it fixed.

Definitely bug on feinstaub. They don't really use that source code, they work separate from what is on github.

[karimkawambwa]

@kilemensi I'll leave the tests, and push the fix to feinstaub then once that is in and the tests pass here, then we will be good on the route.

[karimkawambwa]

• [x] Merged https://github.com/opendata-stuttgart/feinstaub-api/pull/22
• [x] Rerun travis tests

cc @kilemensi

[kilemensi]

Cool @karimkawambwa but lets add a proper description on https://github.com/opendata-stuttgart/feinstaub-api/pull/22. You may then give them a soft ping on the slack channel to look at it.

[kilemensi]

PS: It's also better for the PRs to come from our sensors.AFRICA-AQ-api repo @karimkawambwa.

[karimkawambwa]

PS: It's also better for the PRs to come from our sensors.AFRICA-AQ-api repo @karimkawambwa.

Our repo has a mix of things we don't want to get into feinstaub so I usually have to checkout last commit on feinstaub master and base my changes of that.

[kilemensi]

I thought we either sent PRs back to feinstaub or moved the changes here @karimkawambwa

But we usually keep one branch in a repo completely in sync with upstream so that we can create PRs from it e.g. in CfALabs/wazimap we have openup/master which is kept in sync with OpenUpSA/wazimap master.

[karimkawambwa]

I thought we either sent PRs back to feinstaub or moved the changes here @karimkawambwa

But we usually keep one branch in a repo completely in sync with upstream so that we can create PRs from it e.g. in CfALabs/wazimap we have openup/master which is kept in sync with OpenUpSA/wazimap master.

We can create a new fork of feinstaub since we are going to discontinue sensors.AFRICA-AQ-api

[kilemensi]

We can create a new fork of feinstaub since we are going to discontinue sensors.AFRICA-AQ-api

I agree @karimkawambwa, this is the plan. After having sensors.AFRICA-api in production for a sprint or two, we'll discontinue with sensors.AFRICA-AQ-api and create a fresh fork... But in the meantime, we still need to get our work done through it.