search

RADAR-base / RADAR-Questionnaire

Questionnaire mobile application (Active App) for RADAR-base
https://radar-base.org/
Apache License 2.0
17 stars 15 forks source link

Send Kafka records in batches #1682

Closed mpgxvii closed 1 year ago

mpgxvii commented 1 year ago
blootsvoets commented 1 year ago

LGTM. If you're still running into issues, consider sending the data as a binary recordset: https://github.com/RADAR-base/RADAR-Schemas/edit/master/commons/kafka/record_set.avsc. For an example of an encoder, see https://github.com/RADAR-base/radar-commons/blob/master/radar-commons/src/main/java/org/radarbase/producer/rest/BinaryRecordContent.kt You can send this binary data by using header Content-Type: application/vnd.radarbase.avro.v1+binary.

Another possibility it to compress the data before sending, either with gzip or lzfse compression, however I don't see any cordova libraries to achieve this. If it would be possible, then you could compress either the json or binary data and send the compressed data with header Content-Encoding: gzip or lzfse or deflate.

mpgxvii commented 1 year ago

LGTM. If you're still running into issues, consider sending the data as a binary recordset: https://github.com/RADAR-base/RADAR-Schemas/edit/master/commons/kafka/record_set.avsc. For an example of an encoder, see https://github.com/RADAR-base/radar-commons/blob/master/radar-commons/src/main/java/org/radarbase/producer/rest/BinaryRecordContent.kt You can send this binary data by using header Content-Type: application/vnd.radarbase.avro.v1+binary.

Another possibility it to compress the data before sending, either with gzip or lzfse compression, however I don't see any cordova libraries to achieve this. If it would be possible, then you could compress either the json or binary data and send the compressed data with header Content-Encoding: gzip or lzfse or deflate.

Okay I see, will try them. Thanks so much!