koordinates / kart

Distributed version-control for geospatial and tabular data
https://kartproject.org
Other
525 stars 41 forks source link

Error importing dataset to postgis #529

Closed timlinux closed 2 years ago

timlinux commented 2 years ago

Hi

New to kart here, trying out my first firs activities, I tried to set up postgis as my data repo. I was able to import data but therafter things go wrong:

 timlinux  crest  ../Kart/osgs-kart  main  kart status
kart Traceback (most recent call last):
  File "kart_cli.py", line 4, in <module>
  File "kart-0.10.7-py3.7.egg/kart/cli.py", line 340, in entrypoint
  File "click/core.py", line 829, in __call__
  File "click/core.py", line 782, in main
  File "kart-0.10.7-py3.7.egg/kart/cli.py", line 163, in invoke
  File "click/core.py", line 1259, in invoke
  File "click/core.py", line 1066, in invoke
  File "click/core.py", line 610, in invoke
  File "click/decorators.py", line 21, in new_func
  File "kart-0.10.7-py3.7.egg/kart/status.py", line 41, in status
  File "kart-0.10.7-py3.7.egg/kart/status.py", line 83, in get_working_copy_status_json
  File "kart-0.10.7-py3.7.egg/kart/working_copy/base.py", line 485, in diff_to_tree
  File "kart-0.10.7-py3.7.egg/kart/working_copy/base.py", line 510, in diff_db_to_tree
  File "kart-0.10.7-py3.7.egg/kart/working_copy/base.py", line 525, in diff_db_to_tree_meta
  File "kart-0.10.7-py3.7.egg/kart/working_copy/base.py", line 631, in meta_items
  File "kart-0.10.7-py3.7.egg/kart/sqlalchemy/adapter/base.py", line 155, in all_v2_meta_items
  File "kart-0.10.7-py3.7.egg/kart/utils.py", line 25, in wrapper
  File "kart-0.10.7-py3.7.egg/kart/sqlalchemy/adapter/postgis.py", line 217, in all_v2_meta_items_including_empty
KeyError: 'srtext'
[64069] Failed to execute script kart_cli
error: index uses kart extension, which we do not understand
fatal: index file corrupt
error: index uses kart extension, which we do not understand
fatal: index file corrupt
error: index uses kart extension, which we do not understand
fatal: index file corrupt

The preceding operations I did were:

mkdir osgs-kart
cd osgs-kart/
kart init --workingcopy=postgresql://XXXX:YYYY@osgs.YYYY.com/kart/public?sslmode=require
kart import /home/timlinux/Syncthing/StLucia/StLucia/saintluciagis.gpkg 

I am using kart on fedora

 timlinux  crest  ../Kart/osgs-kart  main  kart --version
Kart v0.10.7, Copyright (c) Kart Contributors
» GDAL v3.3.2; PROJ v6.3.0
» PyGit2 v1.7.0; Libgit2 v1.3.0; Git v2.33.GIT
» SQLAlchemy v1.4.11; pysqlite3 v2.6.0/v3.31.1; SpatiaLite v5.0.1; Libpq v10.0.12
» SpatialIndex v1.9.3
error: index uses kart extension, which we do not understand
fatal: index file corrupt
error: index uses kart extension, which we do not understand
fatal: index file corrupt
error: index uses kart extension, which we do not understand
fatal: index file corrupt
craigds commented 2 years ago

Thanks for the issue.

  1. re the main issue, could you show the output of select * from geometry_columns in that database?
  2. That index file corrupt stuff might be coming from your shell $PS1 - if you can paste the value of your $PS1 here too it might help diagnose what's happening.
timlinux commented 2 years ago

@craigds thank you for your response. See notes below for the info you requested:


 timlinux  crest  /tmp  psql kart
psql (13.4)
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

kart=# select * from geometry_columns;
 f_table_catalog | f_table_schema |      f_table_name       | f_geometry_column | coord_dimension | srid |     type     
-----------------+----------------+-------------------------+-------------------+-----------------+------+--------------
 kart            | kart           | administrative_boundary | geom              |               2 |    0 | MULTIPOLYGON
(1 row)

kart=# 

$PS1 output below

 timlinux  crest  /tmp  $PS1
bash: \[\033]0;${USER}@${HOSTNAME}:: command not found...

Not sure which command is not found...

craigds commented 2 years ago

thanks. It looks like kart's expecting to always have a non-zero srid in the typemod for the geometry column. But it's not required by postgis and we can probably parse the geometry constraint instead of relying on the typmod.

re the PS1, I think your shell is trying to execute the contents of PS1 above, hence the command not found. Try echo $PS1 and it should give you something ugly but more usable. e.g. mine looks like:

$ echo $PS1
\[\e]0;\u@\h: \w\a\]\[\e[01;32m\]\u@\h\[\e[0m\]:\[\e[1;34m\]\w\[\e[0;31m\]$(__git_ps1 " (%s)")\[\e[0m\] \$
rcoup commented 2 years ago

@timlinux possible to do kart meta get as well and paste the output here? I think somehow we might have imported a dataset without a CRS defined.

timlinux commented 2 years ago

Ok for the shell:

\[\033]0;${USER}@${HOSTNAME}: ${SSP_PWD}\007\]\n\[\] timlinux \[\]\[\]\[\] crest \[\]\[\]\[\] /tmp \[\]\[\] \[\e[0;38;5;45;49;1m\]

Note that I have a pimped out shell like this:

image

More details from my laptop setup notes here: https://gist.github.com/timlinux/dfcc773be7db435a8a79b8253de69b1b#bash-prompt-theming

Metaget output attached:

metaget.txt

I forgot to mention when I initially reported the issue that this error message seems to arise in git sometimes e.g.

http://techies-world.com/git-error-index-uses-extension-which-we-do-not-understand-fatal-index-file-corrupt/

When I tried the resolution they propose for git it again triggers a python error (probably linked to the original issue here)

image

rcoup commented 2 years ago

So there's no CRS defined/detected in your original imported data — and that leads to the PostGIS working copy side of it having issues (which is a bug).

Workaround is to set a srs_id value into saintluciagis.gpkg in the gpkg_contents & gpkg_geometry_columns tables, then reimport your data into a Kart repository. It should flow smoothly through to PostGIS after that.


Regarding the error: index uses kart extension, which we do not understand; fatal: index file corrupt messages:

Kart doesn't use an index in the same way Git does, and people doing git add / git commit / etc will do somewhat bad things. So we drop a dummy Git index in that is marked with an extension, but regular Git doesn't know about the extension. When your PS1/console invokes git to figure out something to display, I think it's attempting to query the index state (maybe for dirty-ness?), and so you get an error message. It's a bit of a trade-off between being able to reuse existing Git console helpers around branches/etc and Kart not being identical to Git. I think the first line is possibly inevitable, the second line maybe we can work to get removed.

craigds commented 2 years ago

Regarding the error: index uses kart extension, which we do not understand; fatal: index file corrupt messages:

ok I tracked this via synth-shell to this code which basically runs git branch, and if it doesn't error, then runs git status (a few times). The latter checks the git index, hence the kart extension warnings.

I don't see anything we can do about this, other than perhaps submit a patch to synth-shell to make it cleverer (but it's not doing anything bad)

craigds commented 2 years ago

but back to the main problem:

So there's no CRS defined/detected in your original imported data — and that leads to the PostGIS working copy side of it having issues (which is a bug).

I had to decide which part of that sentence points to a bug. Is the bug in the postgis working copy, not supporting datasets which have no CRS? Or is the bug in the importer not rejecting datasets that have no CRS?

the docs suggest that Kart should support datasets with no CRS, so I guess the bug is in the postgis working copy 👍

craigds commented 2 years ago

@timlinux I suppose the other related question is: does your source dataset have a CRS? Perhaps kart import failed to detect it for some reason. If so we should fix that too.