search

ACED-IDP / aced_etl_pod

etl worker pod
MIT License
1 stars 1 forks source link

Upgrade python version to 3.12 #17

Closed bwalsh closed 6 months ago

bwalsh commented 6 months ago

As an ACED user, when I'm submitting jobs, it would be useful to have quick response time.

https://github.com/ACED-IDP/aced_etl_pod/blob/main/etl-job/Dockerfile#L2

Can we update to 3.12 and test.

matthewpeterkort commented 6 months ago

https://github.com/ACED-IDP/aced_etl_pod/commit/5b63db50c40326de23c9afcb34f781f6f64d3eb1

Looks like icebergschematools -> pyjq doesn't support Python >3.10

Glom is an alternative better maintained package that could be used instead of pyjq in the 1 file that it is used in iceberg-schema-tools https://github.com/mahmoud/glom

Going to take a hack at trying to replace this library with glom

4.299 Building wheels for collected packages: pyjq 4.299 Building wheel for pyjq (pyproject.toml): started 19.07 Building wheel for pyjq (pyproject.toml): finished with status 'error' 19.12 error: subprocess-exited-with-error 19.12
19.12 × Building wheel for pyjq (pyproject.toml) did not run successfully. 19.12 │ exit code: 1 19.12 ╰─> [1110 lines of output] 19.12 :3: DeprecationWarning: lib2to3 package is deprecated and may not be able to parse Python 3.10+

PR out for pyjq -> glom: https://github.com/bmeg/iceberg-schema-tools/pull/31/

bwalsh commented 6 months ago

@matthewpeterkort if all is complete with 3.12 upgrade, can you close this issue?