search

scramjetorg / transform-hub

Scramjet Transform Hub (STH) is a runtime supervisor that can run data processing programs called Sequences and manage local resources on any Linux server, Docker on small edge servers, and even large-scale Kubernetes clusters in the cloud or datacenters. It connects to Scramjet Spaces in Scramjet Cloud Platform.
GNU Affero General Public License v3.0
67 stars 8 forks source link

Only one producer can consumed #1024

Closed gzukowski closed 9 months ago

gzukowski commented 10 months ago

Describe the bug Running 2 sequences, which provide info at the topic, only the first producer can be consumed. This doesnt happen at 0.34

Steps to Reproduce issue.zip

  1. Run sth
  2. Deploy consumer
  3. Deploy first producer
  4. Deploy second producer

Expected behavior It should consume the whole stream.

Version (please complete the following information):

alicja-gruzdz commented 9 months ago

Thanks for reporting. Starting from STH version 0.39.0, we have implemented a queuing mechanism for sending data to topics. This decision was driven by the observation that when two sources simultaneously write to the same topic, data can become interleaved and corrupted. To address this, we now allow only one provider to write to a topic at any given time. When a provider sends a POST request to a topic, it is recommended to include the header Expect: 100-continue. Upon receiving this, STH responds with an HTTP 100 Continue status, indicating readiness to receive the topic data payload. Related PR: https://github.com/scramjetorg/transform-hub/pull/1010