BritishGeologicalSurvey / etlhelper

ETL Helper is a Python ETL library to simplify data transfer into and out of databases.
https://britishgeologicalsurvey.github.io/etlhelper/
GNU Lesser General Public License v3.0
105 stars 25 forks source link

Add API posting recipe #88

Closed volcan01010 closed 4 years ago

volcan01010 commented 4 years ago

Description

This merge requests adds a new recipe to README.md. It shows how to post data to a REST API. The script is modified from code we wrote to post data to the SensorThingsAPI.

To test

This is not a code change, so the purpose of the merge review is to see if there can be improvements to the wording. However, the script does work if you want to test it.

  1. Spin up an ElasticSearch container docker run -d --rm --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:7.9.2
  2. Create a file called db.py that creates a DbParams class called ORACLE_DB which corresponds to the BGS development database
  3. Save the example code as copy_samples.py
  4. Replace "samples" in the select query with the Oracle dictionary of your choice
  5. Increase the time range in the __name__ == "__main__" block
  6. Setup the Oracle drivers and password
  7. Run python copy_samples.py

You can modify the logging levels in the bottom block and see how they affect the output.

Check results:

import requests
import json

API_URL = "http://localhost:9200"

headers = {'Content-Type': 'application/json'}
search = {"query": { "match_all": {} }}
result = requests.get(API_URL + '/samples/_search', headers=headers, data=json.dumps(search))
print(result.json())
ximenesuk commented 4 years ago

This all looks good. I adapted the recipe easily to run against a very different table with no problems.