search

nickchomey / conduit-connector-surrealdb

2 stars 2 forks source link

Feature: Use Go Routines for parsing/processing the incoming batched records #5

Open nickchomey opened 1 month ago

nickchomey commented 1 month ago

Feature description

Each record needs to be parsed and processed against various things - check for and apply relationship schema, etc...

Right now it takes a few milliseconds to do on a few thousand records, but if there's a larger snapshot it would be beneficial to do it in goroutines.

Also just a good exercise to learn those better

nickchomey commented 3 weeks ago

as it stands, this doesnt even really matter - the bottleneck tends to be in the source connector and the surrealdb crud commands. With batching, this connector literally sits waiting idle for the records to come in from the source connector/pipeline.