atviriduomenys / spinta

Spinta is a framework to describe, extract and publish data (a DEP Framework).
MIT License
10 stars 4 forks source link

Shorten long property names for PostgreSQL #553

Open sirex opened 6 months ago

sirex commented 6 months ago

PostgreSQL has name limit, that applies to all names, including table and column names:

https://github.com/atviriduomenys/spinta/blob/5efe352899f72a0fec1f315ff0cd068569b7734d/spinta/backends/postgresql/constants.py#L1-L4

Currently there is a function, to shorten table names:

https://github.com/atviriduomenys/spinta/blob/5efe352899f72a0fec1f315ff0cd068569b7734d/spinta/backends/postgresql/helpers/__init__.py#L17-L24

But this shortening is not used for column names:

https://github.com/atviriduomenys/spinta/blob/5efe352899f72a0fec1f315ff0cd068569b7734d/spinta/backends/postgresql/helpers/__init__.py#L7-L14

Which should be fixes, to also apply shortening for column names too, because we get property names like this:

europos_individualaus_vartojimo_islaidu_pagal_paskirti_klasifikatorius_ecoicop_id

Which is 81 symbols long.

Also we need to check, how many long names we have and if there are many, then we need to implement one time migration, to fix long names.

A manifest with long names:

https://github.com/atviriduomenys/manifest/blob/985447f1e926073b2cbdd2636596b7f2340b2401/datasets/gov/lsd/statistika/svki_mok_tarifai.csv#L8-L9