dimitri / pgloader

Migrate to PostgreSQL in a single command!
http://pgloader.io
Other
5.45k stars 548 forks source link

Exclude default values for specific columns on a table #1518

Open sorvis opened 1 year ago

sorvis commented 1 year ago

I'm trying to load from MSSQL and seeing different columns that have a custom function for the default value or unknown data type. Is there an approach in pgloader where I could have it override that default value for the data type and use the destination db's default value instead? Or perhaps is there a way to just exclude that column? I see there is a way to exclude a table, but did not see in the docs for a way to exclude a column.

```
pgloader version "3.6.9"
compiled with SBCL 2.3.4
```

Incude a self-contained pgloader command file.

If you're loading from a database, consider attaching a database dump to
your issue. For MySQL, use `mysqldump`. For SQLite, just send over your
source file, that's easy. Maybe be the one with your production data, of
course, the one with just the sample of data that allows me to reproduce
your bug.

When using a proprietary database system as a source, consider creating
a sample database on some Cloud service or somewhere you can then give
me access to, and see my email address on my GitHub profile to send me
the credentials. Still open a public issue for tracking and as
documentation for other users.
--
-- EDIT THIS FILE TO MATCH YOUR BUG REPORT
--

LOAD DATABASE

     FROM mssql://***:***@***/***CSI

     INTO postgresql://***:***@***:25060/***CSI?sslmode=allow

alter schema 'dbo' rename to 'public';
2023-08-17T12:10:23.009514+01:00 LOG pgloader version "3.6.9"
2023-08-17T12:10:23.011587+01:00 LOG Data errors in '/private/tmp/pgloader/'
2023-08-17T12:10:23.011628+01:00 LOG Parsing commands from file #P"/Users/***/projects/datamart/csi.load"
2023-08-17T12:10:23.673354+01:00 LOG Migrating from #<MSSQL-CONNECTION mssql://***@***:1433/***CSI {1006512C53}>
2023-08-17T12:10:23.673604+01:00 LOG Migrating into #<PGSQL-CONNECTION pgsql://***@***:25060/***CSI {1006512CD3}>
Max connections reached, increase value of TDS_MAX_CONN
2023-08-17T12:10:26.593467+01:00 WARNING Failed to find table "fn_Split" in schema "dbo" for index "PK__fn_Split__DC501A78A7EEB2D8", skipping the index
2023-08-17T12:10:26.593776+01:00 WARNING Failed to find table "tax_getcode_fn" in schema "dbo" for index "PK__tax_getc__43732E6A68EF33B4", skipping the index
2023-08-17T12:10:26.593984+01:00 WARNING Failed to find table "tax_getcode_fn" in schema "dbo" for index "PK__tax_getc__43732E6A68EF33B4", skipping the index
2023-08-17T12:10:26.594081+01:00 WARNING Failed to find table "tax_getcode_fn" in schema "dbo" for index "PK__tax_getc__43732E6A68EF33B4", skipping the index
2023-08-17T12:10:34.839294+01:00 ERROR Database error 42883: function uuid_generate_v4() does not exist
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
QUERY: CREATE TABLE public.allocation
(
  id             bigserial not null,
  alloc_type     text not null,
  alloc_type_id  int not null,
  ref_type       text not null,
  ref_num        text not null,
  ref_line       int not null,
  item           text not null,
  acct           text not null,
  unit1          text default NULL,
  unit2          text default NULL,
  qty            numeric not null,
  dollar_amt     numeric default NULL,
  weight         numeric default NULL,
  noteexistsflag smallint not null default '0',
  inworkflow     smallint not null default '0',
  createdate     timestamptz not null default CURRENT_TIMESTAMP,
  createdby      text not null default 'sa',
  recorddate     timestamptz not null default CURRENT_TIMESTAMP,
  updatedby      text not null default 'sa',
  rowpointer     uuid not null default uuid_generate_v4()
);
2023-08-17T12:10:34.913575+01:00 FATAL Failed to create the schema, see above.
2023-08-17T12:10:34.913793+01:00 LOG report summary reset
       table name     errors       rows      bytes      total time
-----------------  ---------  ---------  ---------  --------------
  fetch meta data          0        581                     3.494s
   Create Schemas          0          0                     0.053s
 Create SQL Types          0          0                     0.117s
    Create tables          0          0                     0.000s
-----------------  ---------  ---------  ---------  --------------
-----------------  ---------  ---------  ---------  --------------