[question] Is the schema singer message supposed to be emitted multiple times for each stream?

[balmasi]

I noticed that running the tap on some streams results in multiple schema messages being output for the same stream.

For example, when extracting from the "Folder" stream, a SCHEMA message is emitted with every API call.

I know that there's nothing in the singer schema that says this is not allowed, but I think most target's do not expect this behaviour and this can have unintended consequence of running schema management code upon every record (or worse).

I noticed this issue cause my custom GCS creates a new file every time there's a SCHEMA message and I was ending up with 1000s of files in GCS for the same stream.

Is this a bug or expected behaviour?

[visch]

I noticed that running the tap on some streams results in multiple schema messages being output for the same stream.

For example, when extracting from the "Folder" stream, a SCHEMA message is emitted with every API call.

I know that there's nothing in the singer schema that says this is not allowed, but I think most target's do not expect this behaviour and this can have unintended consequence of running schema management code upon every record (or worse).

I noticed this issue cause my custom GCS creates a new file every time there's a SCHEMA message and I was ending up with 1000s of files in GCS for the same stream.

Is this a bug or expected behaviour?

Thank you very much for putting this issue in!

Could you show the 1000's of files you're getting, is it just the folder stream or is it others as well. Generally this shouldn't happen, but there are scenarios where it could. There is some optimization we could do to not send as many schema messages it'd be very helpful to see which streams are causing this!

[visch]

If you could provide some examples we can reopen this!