search

denisenkom / go-mssqldb

Microsoft SQL server driver written in go language
BSD 3-Clause "New" or "Revised" License
1.82k stars 500 forks source link

"Reduce use of goroutines" has broken some functionality #619

Open arjitj2 opened 3 years ago

arjitj2 commented 3 years ago

Describe the bug Something about this commit (https://github.com/denisenkom/go-mssqldb/commit/162e654480e2bffeaeab32d1dca87820df58a840) has broken functionality when using this library in conjunction with Ratchet (https://github.com/dailyburn/ratchet)

Best description of the bug I can give is that it's breaking Ratchet's understanding of queries being executed and the data coming out of it.

The introduction of this new commit gives me error="pq: syntax error at or near \")\"" when attempting to do the load aspect of the ETL into postgres, with MSSQL as the extract.

Also, data from one extract is now leaking into the load of the next query in the pipeline.

Not sure what might have gone wrong, my knowledge of Go isn't very deep. For now, I'm manually pointing myself to commit 2b7eb882b6953bc873544823e68843e09a64a0c2.

denisenkom commented 3 years ago

Can you provide a reproducible example?

arjitj2 commented 3 years ago

Hard to provide a good example as it's so dependent on Ratchet. Will do the best I can:

transform := processors.NewPassthrough()

        layout, err = ratchet.NewPipelineLayout(
            ratchet.NewPipelineStage(
                ratchet.Do(extract).Outputs(transform),
            ),
            ratchet.NewPipelineStage(
                ratchet.Do(transform).Outputs(replicationRoleTask),
            ),
            ratchet.NewPipelineStage(
                ratchet.Do(replicationRoleTask).Outputs(load),
            ),
            ratchet.NewPipelineStage(
                ratchet.Do(load),
            ),
        )

pipeline = ratchet.NewBranchingPipeline(layout)

I use this mssql driver to read data in the extract step. This probably won't make sense to you unless you understand the ratchet library.

This pipeline is working great before the commit I noted above. After the breaking commit, the load still happens, but the data appears to be in the wrong order and/or corrupted. When I do the load step into postgres, I get issues saying I missed a parentheses, or that a column value from the last table is trying to make it into the current table.