oeg-upm / gtfs-bench

GTFS-Madrid-Bench: A Benchmark for Knowledge Graph Construction Engines
https://doi.org/10.5281/zenodo.3574492
Apache License 2.0
17 stars 12 forks source link

MySQL "LOCAL INFILE" import #122

Closed daniel-dona closed 3 years ago

daniel-dona commented 3 years ago

Currently, the (My)SQL distribution is based on the CSV one plus a bunch of SQL DDL sentences to create and populate a database.

The use of "LOAD DATA LOCAL INFILE" is not standard SQL nor possible by default on every MySQL/MariaDB server.

Enabling this should be documented, based on this reference:

https://dev.mysql.com/doc/refman/8.0/en/load-data-local-security.html

daniel-dona commented 3 years ago
If LOCAL is specified, the file is read by the client program on the client host and sent to the server. The file can be given as a full pathname to specify its exact location. If given as a relative pathname, the name is interpreted relative to the directory in which the client program was started.

When using LOCAL with LOAD DATA, a copy of the file is created in the directory where the MySQL server stores temporary files. See Section B.3.3.5, “Where MySQL Stores Temporary Files”. Lack of sufficient space for the copy in this directory can cause the LOAD DATA LOCAL statement to fail. 

̣̣The only relevant part is the client side, no need to modify anything in the server side to use the CSV files as source.

It seems that the client does not process the CSV file but transfer the file to the server and then the server handles the file. So, additional configuration on the server side may be necessary.

But at the same time, it worked out of the box for me in Ubuntu 18.04...

dachafra commented 3 years ago

we can add something in the utils folder and close this issue ;-)

jatoledo commented 3 years ago

I give some recommendations.

docker exec -i container-mysql-name sh -c 'exec mysql --local-infile -uroot -ppassword' < mysql_schema.sql

mysql --local-infile -u root -p < mysql_schema.sql