embulk / embulk-output-jdbc

MySQL, PostgreSQL, Redshift and generic JDBC output plugins for Embulk
Other
88 stars 87 forks source link

Update embulk 0.9 #269

Closed hiroyuki-sato closed 4 years ago

hiroyuki-sato commented 4 years ago

Hello, @hito4t This PR implement embulk/embulk#266.

Could you take a look when you get a chance? I'll use gradle-embulk-plugin in a future.

I want to update PostgreSQL and MySQL JDBC Driver after merge this PR.

hito4t commented 4 years ago

@hiroyuki-sato Thank you! I'll review as soon as possible.

hiroyuki-sato commented 4 years ago

Hello, @hito4t Thank you for your comment.

The core team (@dmikurube ) told me that v0.10 will change much around a driver_path. twitter Japanese So, I'll wait v0.10 development.

I already try MySQL connector/J 8 and MariaDB Connector/J. It need some modification, but It may work. https://github.com/hiroyuki-sato/embulk-output-jdbc/commits/topic/update-mysql-jdbc

If we need to update PostgreSQL and MySQL JDBC driver sooner, I'll create another PR that just updates the driver version. (But some options deprecated in Connector/J 8. ex. useLegacyDatetimeCode for example)

hito4t commented 4 years ago

Some tests failed in my environment. Now I'm examining.

hito4t commented 4 years ago

Some tests failed in my environment. Now I'm examining.

Some tests failed because of the following issue. https://github.com/embulk/embulk-standards/issues/39

I'm trying to find a workaround.

hito4t commented 4 years ago

Some tests failed because of the following issue. embulk/embulk-standards#39

I'm trying to find a workaround.

Path#toRealPath will resolve the problem.

hito4t commented 4 years ago

The test of embulk-output-db2 has timestamp data with picoseconds (2016-09-11 12:34:45.123456789012). When the timestamp parser of embulk 0.8 parses it, it will be parsed down to microseconds and nanoseconds are truncated (namely, parsed as 2016-09-11 12:34:45.123456). But when the timestamp parser of embulk 0.9 parses it, the value under seconds will be truncated (namely, parsed as 2016-09-11 12:34:45.000000).

That difference causes failures of some tests.

This problem will be resolved by cutting values under microseconds in the test data (namely, 2016-09-11 12:34:45.123456).

hiroyuki-sato commented 4 years ago

@hito4t Thank you for testing this PR.

As far as I know, this timestamp parser behavior isn't a known issue. Shall we investigate this timestamp parser problem at first, After that, Will we consider how to change the code

At first, I created simple reproduce codes. It may OK, but I need print under milliseconds value. https://github.com/hiroyuki-sato/embulk-support/tree/master/20200327_output-jdbc-timestamp

hiroyuki-sato commented 4 years ago

@hito4t Created the Issue. https://github.com/embulk/embulk/issues/1235

hito4t commented 4 years ago

@hiroyuki-sato Thank you! I found that nanoseconds are correctly parsed by both older parser and newer parser. I've cut picoseconds in test data because they were not actually used in test.

hiroyuki-sato commented 4 years ago

Thank you!