Country.tsv missing

[coreice]

I've been playing around with the scripts today but converting the IFF files seems not to work. I've been using the IFF files of today (14-10-2016) and tested it with downloading and unzipping manually and via the new-iff.sh script in /contrib/ndov. I guess I'm doing something wrong but I cannot figure out what.

./iff-converter.py INFO:root:Parsing IFF delivery INFO:root:Received file from NS for period 2016-07-04 to 2017-08-06 (IFF Standaard uit RITS) Traceback (most recent call last): File "./iff-converter.py", line 431, in main() File "./iff-converter.py", line 428, in main parse_delivery() File "./iff-converter.py", line 390, in parse_delivery sql_countries(countries) File "./iff-converter.py", line 277, in sql_countries simple_dict_writer('country', ['code', 'inland', 'name'], countries) File "./iff-converter.py", line 55, in simple_dict_writer f = codecs.open(OUTPUT_PATH+filename+'.tsv', 'w', 'UTF-8') File "/usr/lib/python2.7/codecs.py", line 881, in open file = builtin.open(filename, mode, buffering) IOError: [Errno 2] No such file or directory: './cache/iff_parsed/country.tsv'

Thanks in advance!

[geertw]

Hmm, that's not working as it should be, but I could reproduce it on a blank installation on my system and it this error seems to be caused by a missing directory cache/iff_parsed.

Is the problem solved when you create a directory iff_parsed in the cache directory? I'll add some checks to the new-iff.sh script to check whether this directory exists.

[geertw]

Also adding some checks to new-iff.sh to work in every directory

[geertw]

@coreice, I've made some changes to the new-iff.sh script. Can you try again with the updated version? You can safely remove cache/ns-latest.zip and cache/ns-old.zip if the script says that the dataset is equal.

[coreice]

@geertw Thanks for your effort 👍. There is still a problem left. After a fresh clone of rdt-serviceinfo I have tried running new-iff.sh:

~/rdt-serviceinfo/contrib/ndov $ ./new-iff.sh ./new-iff.sh: line 4: realpath: command not found mkdir: cannot create directory /cache': Permission denied mkdir: cannot create directory/cache': Permission denied /cache/ns-latest.zip: No such file or directory diff: /cache/ns-latest.zip: No such file or directory diff: /cache/ns-old.zip: No such file or directory New dataset! rm: cannot remove /cache/dataset/*': No such file or directory unzip: cannot find or open /cache/ns-latest.zip, /cache/ns-latest.zip.zip or /cache/ns-latest.zip.ZIP. rm: cannot remove/cache/iff_parsed/*.tsv': No such file or directory ./new-iff.sh: line 32: /iff-converter.py: No such file or directory ./new-iff.sh: line 33: /iff-loader.py: No such file or directory

The directories should have the right permissions and I'm not switching users or something. But then I saw that the script wants to create the cache directory in the root directory of the system instead of ~/rdt-serviceinfo :). I guess this is a small bug?

[geertw]

Hmm, that is strange. Which distro are you using?

[coreice]

Raspbian 7 (Wheezy).

[geertw]

Hi, sorry it took a while to respond. I assumed that realpath is installed by default on Wheezy but turns out it's not.

Can you try if everything works correctly after installing realpath using apt-get install realpath ? That would probably solve all other error messages as well.

[coreice]

Thanks for your reply. Realpath was indeed missing.

./new-iff.sh diff: /home/pi/rdt-serviceinfo/cache/ns-old.zip: No such file or directory New dataset! rm: cannot remove /home/pi/rdt-serviceinfo/cache/dataset/*': No such file or directory Archive: /home/pi/rdt-serviceinfo/cache/ns-latest.zip inflating: /home/pi/rdt-serviceinfo/cache/dataset/changes.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/company.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/connmode.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/contconn.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/country.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/delivery.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/error.tmp inflating: /home/pi/rdt-serviceinfo/cache/dataset/ff_changes_debuginfo inflating: /home/pi/rdt-serviceinfo/cache/dataset/ff_timetbls_spiekbest inflating: /home/pi/rdt-serviceinfo/cache/dataset/footnote.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/gvk.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/halte.tst extracting: /home/pi/rdt-serviceinfo/cache/dataset/hertz.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/iff_config.cfg inflating: /home/pi/rdt-serviceinfo/cache/dataset/kilonet_demo.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/kilonetnew.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/language.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/ns.ttx extracting: /home/pi/rdt-serviceinfo/cache/dataset/ns.ttx.characterset_input extracting: /home/pi/rdt-serviceinfo/cache/dataset/nsrichting.dat extracting: /home/pi/rdt-serviceinfo/cache/dataset/richting.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/srvcdest.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/statconn.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/stations.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/synonym.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/tariefnet.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/timetbls.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/timezone.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/trnsaqst.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/trnsattr.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/trnsmode.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/trnsmqst.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/tweede_klas.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/xchanges.dat inflating: /home/pi/rdt-serviceinfo/cache/dataset/xfootnote.dat rm: cannot remove/home/pi/rdt-serviceinfo/cache/iff_parsed/*.tsv': No such file or directory INFO:root:Parsing IFF delivery INFO:root:Received file from NS for period 2016-07-04 to 2017-08-06 (IFF Standaard uit RITS) INFO:root:Loading TSV files INFO:root:Truncate tables... INFO:root:... changes Traceback (most recent call last): File "/home/pi/rdt-serviceinfo/iff-loader.py", line 89, in main() File "/home/pi/rdt-serviceinfo/iff-loader.py", line 86, in main load_tsv_files() File "/home/pi/rdt-serviceinfo/iff-loader.py", line 43, in load_tsv_files c.execute(q) File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute self.errorhandler(self, exc, value) File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue _mysql_exceptions.ProgrammingError: (1146, "Table 'iff.changes' doesn't exist")

A check in MySQL learned there are no tables created in the database iff.

[coreice]

(PS: Wrong button, didn't want to close the issue.)

[geertw]

There are no tables at all?

[coreice]

Completely empty.

mysql> use iff; Database changed mysql> show tables; Empty set (0.00 sec)

The database connection looks right. Editing the username/password/db name to garbage results in connections errors so that must be fine.

[geertw]

Hm, the script used to load a new IFF dataset does not create a new database. You can either create it manually using the SQL script in doc/create-tables.sql, or modify the script to add the argument --create_tables to iff-loader.py. But this will probably result in errors when run again, so I advise to run the SQL script manually as you'll need to do it only once.

[coreice]

Executing create-tables.sql did the job indeed. Thanks! 👍

[geertw]

Nice!