dimitri / pgloader

Migrate to PostgreSQL in a single command!
http://pgloader.io
Other
5.47k stars 551 forks source link

How to import CSV into postgresql ignore serial column ? #856

Closed suntzu93 closed 6 years ago

suntzu93 commented 6 years ago

Hello, I'm using pgloader verstion : pgloader version "3.5.2" My table :

Column | Type | Collation | Nullable | Default | Storage | Stats target | Description id | integer | | not null | nextval('tbl_db_id_seq'::regclass) | plain | | inserttime | timestamp with time zone | | not null | now() | plain | | id_buildid | smallint | | not null | | plain | | device | text | | | | extended | | device_api | text | | | | extended | | userid | text | | | | extended | | request_name | text | | | | extended | | request_duration | bigint | | | | plain | | count_number | integer | | not null | 1 | plain | | foreground | integer | | not null | 0 | plain | | background | integer | | not null | 0 | plain | | worker_thread | integer | | not null | 0 | plain | | main_thread | integer | | not null | 0 | plain | | status | smallint | | not null | 0 | plain | | flow_tag | smallint | | not null | '-1'::integer | plain | | Partition key: RANGE (inserttime)

My Csv file hasn't the value of "id" column so I wanna Ignore this column when run pgloader. My pgloader command :

pgloader --type csv \ --field "inserttime,id_buildid,device,device_api,userid,request_name,request_duration,count_number,foreground,background,worker_thread,main_thread,status,flow_tag" \ --with "skip header = 1" \ --with "fields terminated by '\t'" \ /home/cpu60020-local/Documents/abc/1539672600000.csv \ postgresql://username:123456@localhost/log?tablename=tbl_db

and I got an error :

ERROR Could not read input: The variable PGLOADER.USER-SYMBOLS::ID is unbound.

So How can I Ignore "id" column ?

suntzu93 commented 6 years ago

I can ignore the id with command :

LOAD CSV FROM '/home/cpu60020-local/Documents/abc/1539672600000.csv' (inserttime,id_buildid,device,device_api,userid,request_name,request_duration,count_number,foreground,background,worker_thread,main_thread,status,flow_tag) INTO postgresql://username:123456@localhost/log?tablename=tbl_db (inserttime,id_buildid,device,device_api,userid,request_name,request_duration,count_number,foreground,background,worker_thread,main_thread,status,flow_tag) WITH truncate, skip header = 1, fields optionally enclosed by '"', fields escaped by double-quote, fields terminated by '\t'