Poetry Installation Update and Dependencies Update

[td928]

350 which was flagged earlier in #347 and #326

This installation update is fairly simple and we have done very similar works on gru qaqc recently. On other other hand, one unexpected complexity is after poetry installation is updated and the existing poetry.lock no longer works. This leads me down in a deep rabbit hole on learning about poetry maintenance and Python dependencies shenanigans. I will share some of my most important learnings here.

poetry install/lock -vv

After reading a bunch on the poetry slow resolve dependency issues, I finally stumbled across this useful command with poetry install -vv. What it does is showing the verbose details for poetry resolving dependencies and some comments from post linked also highlighted that poetry is essentially going through each possible version and figuring out the dependency graph for each version. One can imagine this becomes very expensive if the lower bound version to be very far from the latest. So the trial and error way for me to solve this is

1. run this command with verbose option and then see which dependencies it got hang on. Terminates the process by aborting (control + c) if needed
2. raise the lower bounds for that package close to latest version
3. rerun poetry install to see if it still hangs. After a few tries with this, the dependencies resolving time dropped to around 10 seconds or so.

gdal

updated to latest 3.6.1

jinja2

unrelated but there is also a issue after everything else is reinstalled so it is also updated.

Notes for testing

I run some datasets to test locally whether the functionalities for data library is working properly. For example I was running library archive -n dcp_nta2020 -v 22c

[mbh329]

The test of the dcp_nta2020 data worked on my local machine but going to do some more testing so that the new packages work as expected