hasadna / knesset-data-pipelines

Main repository for Open Knesset project - contains the knesset data scrapers and processing pipelines
https://oknesset.org/
MIT License
14 stars 26 forks source link

Missing tables names from information_schema.tables #245

Closed bobiboMC closed 1 year ago

bobiboMC commented 1 year ago

There are tables that I saw added to the database, such as committees_kns_committeesession and committees_kns_documentcommitteesession, among others, which are not shown in information_schema.tables table .

OriHoch commented 1 year ago

does not reproduce for me, how exactly are you connecting to the DB? What query are you running?

these are my connection details (without the password):

OKNESSET_DB_HOST=212.80.204.81
OKNESSET_DB_PORT=5432
OKNESSET_DB_USER=redash_reader_2
OKNESSET_DB_PASSWORD="******************"
OKNESSET_DB_NAME=postgres

the query I'm running:

select * from information_schema.tables
where table_name in ('committees_kns_documentcommitteesession', 'committees_kns_committeesession')

The result:

table_catalog table_schema table_name table_type self_referencing_column_name reference_generation user_defined_type_catalog user_defined_type_schema user_defined_type_name is_insertable_into is_typed commit_action
postgres public committees_kns_documentcommitteesession BASE TABLE null null null null null YES NO null
postgres public committees_kns_committeesession BASE TABLE null null null null null YES NO null
bobiboMC commented 1 year ago

I was connecting with the same credentials as you provide. I ran the following query: SELECT table_name FROM information_schema.tables WHERE table_schema NOT IN ('pg_catalog', 'information_schema') and also this one: SELECT * FROM information_schema.tables WHERE table_name in ('committees_kns_documentcommitteesession', 'committees_kns_committeesession') For the first query, I didn't see these tables names in the result. For the second query, I got empty array in the result. Now the tables are showing for me for both of the queries, and basically all of the tables are showing correctly. When I ran the queries 4 days ago I couldn't find those tables.

OriHoch commented 1 year ago

do you remember what time it was when it failed? it might be related to our pipeline run

bobiboMC commented 1 year ago

It was around 17:00 to 18:00 on July 5th when I tried it.

OriHoch commented 1 year ago

I made some changes to how the pipelines work and create tables, I think it should fix it, let me know if it happens again