search

seanharr11 / etlalchemy

Extract, Transform, Load: Any SQL Database in 4 lines of Code.
MIT License
556 stars 83 forks source link

Problem for get_nearest_power_of_two #10

Closed deadmanhead1602 closed 7 years ago

deadmanhead1602 commented 7 years ago

I try to migrate oracle DB to mysql and find that Datatype VARCHAR2(2 byte) will convert to VARCHAR(1).

The problem due to get_nearest_power_of_two. 2 of power of two should be 4.

Log trace: ETLAlchemySource (INFO) - (code) VARCHAR ETLAlchemySource (INFO) - Bases: ['STRING'] ETLAlchemySource (INFO) - Converting to -> VARCHAR(1 (maxsize: 2)

seanharr11 commented 7 years ago

@deadmanhead1602 - glad you pointed this out, had a major flaw in the logic here if the maxsize was an even power of 2.

Commit b8b7d3a47c1e7e52d7f0233257fdfa1164f64c98 should fix this, as well as optimize the deduced column size on MySQL for disk-size, memory & caching.

This stackoverflow answer explains why powers of 2 and minimizing the VARCHAR(n) size matters, specifically on MySQL.

Can you pull the latest changes to your repository, install the package to a local virtualenv, and re-run your test above? This way I can have another set of us eyes verify before packaging this in a release. (I need to build some tests into this repository, I know that's lacking...)

Thanks! And let me know if you have issues trying to install directly from the git repo.

seanharr11 commented 7 years ago

@deadmanhead1602 did this work for you?