dimitri / pgloader

Migrate to PostgreSQL in a single command!
http://pgloader.io
Other
5.45k stars 548 forks source link

Direct URL can't be mapped to file system #1606

Open bobir01 opened 3 months ago

bobir01 commented 3 months ago

URL that does not end with simple file link query cant be read to system files data url is changed to public csv file due to risks of privacy, but the core problem is not about file content here ;(


LOAD CSV
    FROM 'https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z' WITH ENCODING utf-8
    INTO postgresql://user:*****@localhost:5432/db
    TARGET TABLE cdm_commerce.dm_basic_osa_hourly

    WITH
        skip header = 1,
        fields terminated by ','
        SET work_mem to '1GB', maintenance_work_mem to '2GB';

BORDET|Bobirjon Mardonov
2024-08-26T13:09:57.012000Z LOG pgloader version "3.6.7~devel"
2024-08-26T13:09:57.012000Z LOG Data errors in '/tmp/pgloader/'
2024-08-26T13:09:57.012000Z LOG Parsing commands from file #P"/home/b_mardonov/csv.load"
KABOOM!
SIMPLE-FILE-ERROR: Can't find the TRUENAME of wild pathname /home/b_mardonov/https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z) (physicalized from /home/b_mardonov/https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z).
An unhandled error condition has been signalled:
   Can't find the TRUENAME of wild pathname /home/b_mardonov/https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z (physicalized from /home/b_mardonov/https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z).

What I am doing here?

Can't find the TRUENAME of wild pathname #1=/home/b_mardonov/https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z (physicalized from #1#).
[data.csv](https://www.stats.govt.nz/assets/Uploads/Annual-enterprise-survey/Annual-enterprise-survey-2023-financial-year-provisional/Download-data/annual-enterprise-survey-2023-financial-year-provisional-size-bands.csv?sv=2023-11-03&st=2024-08-26T11%3A45%3A43Z&se=2024-08-30T11%3A45%3A00Z)
bobir01 commented 3 months ago

this seems very silly issue why there is no option to rename blob downloading ?