search

slingdata-io / sling-cli

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
https://docs.slingdata.io
GNU General Public License v3.0
446 stars 34 forks source link

refactor writeto db to smaller function #399

Closed yokofly closed 1 month ago

yokofly commented 1 month ago

I intend to split as 2 prs for #35 the first part is to refactor the writetodb function, I have run some tests, probably the CI will run a more compatible suite.

I conduct a more direct test to verify the temp table: insert a big table, stop with ctrl+c during the insert, and collect the log this is the whole log difference(between release 1.2.20 and commit ac8802a30), from the log seems all good https://www.diffchecker.com/f4YCxD9W/

flarco commented 1 month ago

Thanks, started to test locally but dealing with another issue. I'm travelling this week so I may not get to it till this weekend or next week.

Also, there are a few instances of the use of g.Error that need fixing.

For example, instead of g.Error("Could not perform upsert from temp: %v", err) could you use g.Error(err, "Could not perform upsert from temp")? It's a wrapper that I use for error logging/stack tracing, if the first param is anerror type, it treats it differently.

yokofly commented 1 month ago

hahaha, let's enjoy the trip first! Will adjust the error wrapper.

yokofly commented 1 month ago

I locally performed some simple tests with clickhouse to clickhouse

flarco commented 1 month ago

Tested, all good 👍 .