earthcubeprojects-chords / chords

EarthCube CHORDS application code
GNU General Public License v2.0
25 stars 13 forks source link

Unable to store data on CHORDS portal using python GET requests #563

Closed jaypotnis closed 3 years ago

jaypotnis commented 3 years ago

I have created a CHORDS portal on AWS using the instructions on https://earthcubeprojects-chords.github.io/chords-docs/gettingstarted/

I am trying to store some data on the portal using a series of python GET requests. But the CHORDS server does not save the data and returns a response 500 for the requests.

The format of my python requests is: gauge_url = f'http://ec2-3-131-30-243.us-east-2.compute.amazonaws.com/measurements/url_create?sensor_id=1&rc={int(rain_class)}&at={int(grid_year)}-{int(grid_month):02d}-{int(grid_day):02d}T{int(grid_hour):02d}:{int(grid_min):02d}:{int(grid_sec):02d}&email=user@gmail.com&api_key=tDF-1uMcyU4Vn-XHLoB2&test' gauge_response = requests.get(url=gauge_url)

Python returns the following error message:

response.txt

Here is the response of the Docker log file for the Python request.

log.txt

Any help on this is appreciated. Thanks.

mdye commented 3 years ago

Hello Jay,

Looking at the log.txt file, it looks like the problem is with the retention policy:

InfluxDB::Error ({"error":"partial write: points beyond retention policy dropped=1"}

I'm guessing you are populating the portal with historical data the is older than what is in the retention policy. The default specified in the CloudFormation template is 52 weeks. Any data older than that is automatically discarded (and data older than the retention policy cannot be inserted.)

I see two possible solutions: 1) Rebuild the portal, selecting the INF (infinite / unlimited) option for the retention policy

2) Log in to your portal vs ssh and manually reset the db retention period. This can be done as follows:

cd /var/lib/chords
python chords_control --config
python chords_control --stop
python chords_control --run

During the config step, hit enter (which selects the existing value for that configuration items) until you get to the DB_RETENTION prompt. Type inf and hit enter until you are through the remaining configuration.

Hopefully after this step and restarting the chords service, the retention period will be updated and you can import your data.

jaypotnis commented 3 years ago

Changing the retention period and restarting solved the issue. Thanks a lot for the help @mdye . Really appreciate it.