search

RedHatInsights / aiops-insights-clustering

Clustering of systems
GNU General Public License v3.0
1 stars 14 forks source link

Read raw chunked CSV #29

Closed tumido closed 5 years ago

tumido commented 5 years ago

As of https://github.com/ManageIQ/aiops-data-collector/pull/16 we may change the way this service consumes input data. This PR makes it ready for such change.

This PR changes the input from reading a JSON:

  1. Instead it waits for get_data() buffer from a streamed raw byte stream and populates a BytesIO.
  2. That's read as a CSV file into a DataFrame.
  3. Then it's the same business as usual.
  4. The job ID is no longer stored within the JSON on input.
  5. Instead we encode it into a HTTP header source_id.