Closed nj1973 closed 1 week ago
Are the trailing spaces in a DDL statement necessary?
One of my tests was to re-use an existing DDL file.
Option error: 'DDL path already exists: /tmp/t3.sql'
Makes me wonder whether there should be a --reuse-ddl-file
option?
Are the trailing spaces in a DDL statement necessary?
Not necessary no, I've not changed the SQL text generation, only redirected it to a file. I suppose we should look to address this though because I wouldn't like trailing spaces in code in my repo.
One of my tests was to re-use an existing DDL file.
Option error: 'DDL path already exists: /tmp/t3.sql'
Makes me wonder whether there should be a
--reuse-ddl-file
option?
I don't think so, I think that is one option too many. We already have too many. We should either abort as above or overwrite the file. I chose abort because it is safer.
Data type modifications are not reflected in the DDL file. For example --variable-string-columns=created
ignored and DDL file still has this column as DATETIME
.
Using the --reuse-backend-table
and --reset-backend-table
options together gives the latter precedence and drops the backend table. I think this combination should be a message to the user to choose the appropriate option only.
Screen output is minimal and shows nothing for short-circuited (i.e. prevented) offloads. Includes, for example, trying to --reuse-backend-table
when it is populated.
If a pre-existing backend table has a different column type to the frontend table, the mismatch is raised, even if an offload override option is used to bring them into alignment.
Are the trailing spaces in a DDL statement necessary?
These have been removed.
Using the
--reuse-backend-table
and--reset-backend-table
options together gives the latter precedence and drops the backend table. I think this combination should be a message to the user to choose the appropriate option only.
Fixed, thanks
Screen output is minimal and shows nothing for short-circuited (i.e. prevented) offloads. Includes, for example, trying to
--reuse-backend-table
when it is populated.
Fixed, thanks
DDL file This PR adds a new option
--ddl-file
that will write CREATE TABLE DDL to a file and exit the Offload command without copying data. The option supports writing the file to local storage or cloud storage. Based on comments on this PR the DDL file also includes DDL to support the--create-backend-db
option.The basic use case is output the DDL for a table that has not yet been offloaded at all. We've plugged it into the existing flow at the part where table DDL is generated, therefore and SQL commands to create backend databases (
--create-backend-database
) or drop existing tables (--reset-backend-table
) will still be visible in the output. Any command that runs in execute mode will be downgraded to non-execute mode.You'll notice in the file changes that there are a lot of changes related to
execute
anddry_run
. That is because I needed to move theexecute
flag fromOrchestrationConfig
toOffloadOperation
. I'd tried to do this previously but was thwarted by Incremental Update. Now that Incremental Update has been removed I was able to make this change. I needed the change because--ddl-file
will switch a command fromexecute=True
toexecute=False
and changing it in config was the wrong thing to do.Offload to existing table This PR also adds support for Offloading to a pre-created table - but only if the table is empty and has no Offload metadata.
If the table has data then the normal
--reset-backend-table
option is required.If the table is empty but has metadata (for example a sysadmin may have truncated the backend table) then the new
--reuse-backend-table
option is required.