search

go-testfixtures / testfixtures

Ruby on Rails like test fixtures for Go. Write tests against a real database
https://pkg.go.dev/github.com/go-testfixtures/testfixtures/v3?tab=doc
MIT License
1.1k stars 80 forks source link

WIP: Feature: support bulk insert to speed insert times #181

Open alexisvisco opened 9 months ago

alexisvisco commented 9 months ago

Description

Overview

This pull request introduces a new feature to the fixture library, enabling bulk inserts for fixture files. Prior to this enhancement, inserts were performed sequentially, leading to potential performance bottlenecks, especially with large datasets. The new functionality allows users to opt for bulk inserts, significantly improving the efficiency of the fixture loading process.

Changes Made

Added a functional option parameter UseBulkInsert() to allow bulk inserting for fixture files.

Modified the implementation to group records with exactly same columns together for efficient bulk inserts when this parameter is set to true.

Also add a free new STRING= like RAW= because I was in trouble this last days due to time conversion for string ids... So with this new "annotation" it solve the problem.

Benefits

Improved performance by enabling bulk inserts.

Maintained flexibility by allowing users to choose between sequential and bulk inserts using the UseBulkInsert() functional option (mostely because I write this for postgres in the first place and I don't know if it works for other databse right now)

Feedback and suggestions for further improvements are welcome!

Not tested right now but this is coming.

andreynering commented 7 months ago

Hi @alexisvisco, thanks for opening the PR!

The change looks promising, indeed. Performance gains are certainly welcome!

I agree to make this a setting so it's not enabled by default.

To make this complete, I'd be awesome to have tests for each database system where it's possible to have bulk loading working. Also, updating the README to contain the new setting would be nice.

alexisvisco commented 7 months ago

Hi @alexisvisco, thanks for opening the PR!

The change looks promising, indeed. Performance gains are certainly welcome!

I agree to make this a setting so it's not enabled by default.

To make this complete, I'd be awesome to have tests for each database system where it's possible to have bulk loading working. Also, updating the README to contain the new setting would be nice.

Yep sure, I was just doubting somebody is anwsering to me so I just fork it, we use my fork to test our application at my company since december.

I would add tests indeed ! :)

slsyy commented 1 month ago

@alexisvisco do you work on this PR anymore?

alexisvisco commented 1 month ago

@alexisvisco do you work on this PR anymore?

Hey ! I do not have time to work on this in the future weeks, but eventually you can add tests, if you have any questions I can reply to you.