search

ODM2 / ODM2DataSharingPortal

A Python-Django web application enabling users to upload, share, and display data from their environmental monitoring sites via the app's ODM2 database. Data can either be automatically streamed from Internet of Things (IoT) devices, manually uploaded via CSV files, or manually entered into forms.
BSD 3-Clause "New" or "Revised" License
31 stars 8 forks source link

Lighter Weight Data Post Request #11

Open horsburgh opened 7 years ago

horsburgh commented 7 years ago

The current POST to send data to the website request is heavy because it contains a lot of information - e.g., tokens, tags, etc. Need to consider a lighter weight post request (or potentially even a get request) that simply has a list of ordered numbers, one for each measured variable.

Issues:

  1. How to make sure the values are always in the right order in the request?
  2. How to match the values to the right result on the back end
  3. How to get some security so we don't get spammed (check to make sure requests are authentic).
SRGDamia1 commented 7 years ago

The Mayfly itself seems to max out at dealing with strings above roughly 3000 characters. Other Arduinos with less beefy processing chips could do less.

horsburgh commented 7 years ago

@SRGDamia1 and @aufdenkampe - in looking at this a little bit, it looks like a POST request in JSON format with all of the UUIDs wouldn't exceed 3000 characters until you had 40+(ish) variables in the post request (depending on how many digits are included in the numeric values for each variable). I captured this issue from our conversations, but is this a size constraint or a speed constraint? If it's size, do we really anticipate more than 40 variables in a single POST request?

aufdenkampe commented 7 years ago

All of this is true, but but savings on data plans might still be a motivation. My 160065 CROSSLANDSPOND - 10 VARIABLES station (http://data.envirodiy.org/sites/6/) averages about 0.74 MB per day, or ~22.5 MB per month. And that is when things are working well (see https://github.com/EnviroDIY/ModularSensors/issues/65).

Regardless, this is still probably a lower priority issue for now, as these POST requests are working for our needs. We can optimize later.

aufdenkampe commented 6 years ago

The present EnviroDIY post request approach uses a significantly higher amount of data bandwidth (3-10x more -- need to quantify) than what we had previously used with Dreamhosters. We had originally been concerned about the capabilities of the radios and Mayfly logger, but the issue now is the actual cost of paying for the SIM card subscription. We can no longer get by on 5-10 MB per month, and we often exceed 25 MB per month. This adds up to a non-trivial expense for a network.

Related to issues #76 & #95.

horsburgh commented 6 years ago

@aufdenkampe - I'm going to backburner this one. We have a working prototype for the data submission POST service. We can revisit this one later if we have time/resources.

To reduce data submission requirements - consider reducing data collection/transmission frequency. 5-minute data is overkill for many (most?) applications. The EnviroDIY site is totally capable of accepting data at this frequency, but if the limitation is the cost of the data connection, then users need to balance data collection frequency needs with available budget.

SRGDamia1 commented 6 years ago

Most of the extra "weight" of the post to EnviroDIY is the GUID's.

aufdenkampe commented 6 years ago

I'm OK with putting this on the back burner for now, and for suggesting 10 or 15 min intervals for those who want to save $$.