Open erichanson opened 9 months ago
Getting close. Running the extension-to-bundle.sql script calls extension_to_bundle() on each extension except meta and bundle, and outputs below.
After the conversion to bundles, I deleted all the bundled schemas and recreated their extensions, which gives a really nice diff between what's in the bundle and what the extension creates.
Remaining issues:
information_schema.parameters
is really useless and awful in this spaceuntracked_row
contains every pg object in bundle and meta. As it should. But this has to be addressed. When a schema or relation is ignored, its non-meta representation doesn't show up in meta, but it's meta row still does. Meta either sees everything or nothing.head_commit_row_with_exists
and head_db_stage_changed
are very slow. The expensive part here is the check to see if the row exists in the working copy. Gotta figure something out.meta.constraint_unique
and meta.constraint_check
views are not being imported yet.meta.type
only supports insert/update/delete of enums and composite types.Null display is "💩".
Timing is on.
Timing is on.
BEGIN
Time: 0.602 ms
CREATE EXTENSION
Time: 31.941 ms
CREATE EXTENSION
Time: 3.067 ms
CREATE EXTENSION
Time: 5.236 ms
CREATE EXTENSION
Time: 81.313 ms
CREATE EXTENSION
Time: 28.216 ms
CREATE EXTENSION
Time: 70.854 ms
SET
Time: 0.524 ms
INSERT 0 12
Time: 0.661 ms
CREATE FUNCTION
Time: 1.242 ms
NOTICE: extension_to_bundle(): Converting extension event to bundle org.aquameta.ext.event
NOTICE: bundle: Committing to org.aquameta.ext.event
NOTICE: bundle: Committing rowset_rows...
NOTICE: bundle: Committing blobs...
NOTICE: bundle: Committing stage_row_fields...
NOTICE: bundle: Creating the commit...
NOTICE: bundle: Updating bundle.head_commit_id...
NOTICE: bundle: Cleaning up after commit...
NOTICE: bundle.checkout(): org.aquameta.ext.event / e20b5add-6d79-464c-871d-a83645529309 @ 2023-11-21 05:26:07.745313 by <NULL>: "initial import"
extension_to_bundle
---------------------
(1 row)
Time: 36065.190 ms (00:36.065)
NOTICE: extension_to_bundle(): Converting extension endpoint to bundle org.aquameta.ext.endpoint
NOTICE: bundle: Committing to org.aquameta.ext.endpoint
NOTICE: bundle: Committing rowset_rows...
NOTICE: bundle: Committing blobs...
NOTICE: bundle: Committing stage_row_fields...
NOTICE: bundle: Creating the commit...
NOTICE: bundle: Updating bundle.head_commit_id...
NOTICE: bundle: Cleaning up after commit...
NOTICE: bundle.checkout(): org.aquameta.ext.endpoint / f570bd07-3253-404f-a87d-3b4914689fd7 @ 2023-11-21 05:26:07.745313 by <NULL>: "initial import"
NOTICE: CHECKOUT EXCEPTION checking out (meta,function,id,"(endpoint,path_to_function_id,{text})"): function endpoint.urldecode_arr(text) does not exist
NOTICE: CHECKOUT EXCEPTION checking out (meta,function,id,"(endpoint,column_list,""{text,text,text,pg_catalog._text,pg_catalog._text}"")"): "column_list" is not a known variable
NOTICE: CHECKOUT EXCEPTION checking out (meta,function,id,"(endpoint,rows_select_function,""{meta.function_id,json}"")"): "mimetype" is not a known variable
NOTICE: CHECKOUT EXCEPTION checking out (meta,function,id,"(endpoint,field_select,{meta.field_id})"): "mimetype" is not a known variable
NOTICE: CHECKOUT EXCEPTION checking out (meta,function,id,"(endpoint,columns_json,""{text,text,pg_catalog._text,pg_catalog._text}"")"): "json" is not a known variable
NOTICE: CHECKOUT EXCEPTION checking out (meta,function,id,"(endpoint,anonymous_rows_select_function,""{text,text,json}"")"): variable "mimetype" does not exist
extension_to_bundle
---------------------
(1 row)
Time: 88972.754 ms (01:28.973)
NOTICE: extension_to_bundle(): Converting extension widget to bundle org.aquameta.ext.widget
NOTICE: bundle: Committing to org.aquameta.ext.widget
NOTICE: bundle: Committing rowset_rows...
NOTICE: bundle: Committing blobs...
NOTICE: bundle: Committing stage_row_fields...
NOTICE: bundle: Creating the commit...
NOTICE: bundle: Updating bundle.head_commit_id...
NOTICE: bundle: Cleaning up after commit...
NOTICE: bundle.checkout(): org.aquameta.ext.widget / 893fcc7a-0e5b-4613-a41b-d607975a8881 @ 2023-11-21 05:26:07.745313 by <NULL>: "initial import"
extension_to_bundle
---------------------
(1 row)
Time: 29897.816 ms (00:29.898)
NOTICE: extension_to_bundle(): Converting extension semantics to bundle org.aquameta.ext.semantics
NOTICE: bundle: Committing to org.aquameta.ext.semantics
NOTICE: bundle: Committing rowset_rows...
NOTICE: bundle: Committing blobs...
NOTICE: bundle: Committing stage_row_fields...
NOTICE: bundle: Creating the commit...
NOTICE: bundle: Updating bundle.head_commit_id...
NOTICE: bundle: Cleaning up after commit...
NOTICE: bundle.checkout(): org.aquameta.ext.semantics / d0d90269-e733-4cf5-ba6c-58cadddcc4e5 @ 2023-11-21 05:26:07.745313 by <NULL>: "initial import"
extension_to_bundle
---------------------
(1 row)
Time: 29187.992 ms (00:29.188)
NOTICE: extension_to_bundle(): Converting extension ide to bundle org.aquameta.ext.ide
NOTICE: bundle: Committing to org.aquameta.ext.ide
NOTICE: bundle: Committing rowset_rows...
NOTICE: bundle: Committing blobs...
NOTICE: bundle: Committing stage_row_fields...
NOTICE: bundle: Creating the commit...
NOTICE: bundle: Updating bundle.head_commit_id...
NOTICE: bundle: Cleaning up after commit...
NOTICE: bundle.checkout(): org.aquameta.ext.ide / e85ad162-0be2-4b61-a18d-3da7f91c84b0 @ 2023-11-21 05:26:07.745313 by <NULL>: "initial import"
extension_to_bundle
---------------------
(1 row)
Time: 5623.233 ms (00:05.623)
NOTICE: extension_to_bundle(): Converting extension documentation to bundle org.aquameta.ext.documentation
NOTICE: bundle: Committing to org.aquameta.ext.documentation
NOTICE: bundle: Committing rowset_rows...
NOTICE: bundle: Committing blobs...
NOTICE: bundle: Committing stage_row_fields...
NOTICE: bundle: Creating the commit...
NOTICE: bundle: Updating bundle.head_commit_id...
NOTICE: bundle: Cleaning up after commit...
NOTICE: bundle.checkout(): org.aquameta.ext.documentation / 6054192c-64a4-48df-a67d-3aab9982603a @ 2023-11-21 05:26:07.745313 by <NULL>: "initial import"
extension_to_bundle
---------------------
(1 row)
Time: 7631.344 ms (00:07.631)
COMMIT
Time: 19.276 ms
Timing on bundle views, with everything installed:
dev@aquameta:~/dev/aquameta/experimental$ cat view-timing.sql | psql -e
select count(*) from _bundle_blob;
count
-------
5620
(1 row)
Time: 243.264 ms
select count(*) from head_commit_field;
count
-------
6400
(1 row)
Time: 26.291 ms
select count(*) from head_commit_row;
count
-------
2060
(1 row)
Time: 2.947 ms
select count(*) from head_commit_row_with_exists;
count
-------
2060
(1 row)
Time: 518.459 ms
select count(*) from head_db_stage;
count
-------
2075
(1 row)
Time: 3995.566 ms (00:03.996)
select count(*) from head_db_stage_changed;
count
-------
15
(1 row)
Time: 3842.206 ms (00:03.842)
select count(*) from not_ignored_row_stmt;
count
-------
52
(1 row)
Time: 54.009 ms
select count(*) from offstage_field_changed;
count
-------
0
(1 row)
Time: 1366.073 ms (00:01.366)
select count(*) from offstage_row_deleted;
count
-------
0
(1 row)
Time: 506.547 ms
select count(*) from offstage_row_deleted_by_relation;
count
-------
0
(1 row)
Time: 485.427 ms
select count(*) from offstage_row_deleted_by_schema;
count
-------
0
(1 row)
Time: 481.131 ms
select count(*) from stage_row;
count
-------
2060
(1 row)
Time: 5.149 ms
select count(*) from stage_row_field;
count
-------
6400
(1 row)
Time: 753.301 ms
select count(*) from trackable_relation;
count
-------
52
(1 row)
Time: 20.232 ms
select count(*) from tracked_row;
count
-------
2075
(1 row)
Time: 14.473 ms
select count(*) from untracked_row;
count
-------
2
(1 row)
Time: 3233.124 ms (00:03.233)
select count(*) from untracked_row_by_relation;
count
-------
2
(1 row)
Time: 3266.709 ms (00:03.267)
select count(*) from untracked_row_by_schema;
count
-------
2
(1 row)
Time: 3253.589 ms (00:03.254)
Aquameta's extensions, except for
meta
andbundle
(the bootstrap), should be converted to bundles. The extension-to-bundle.sql script is the first of such experiments, but it can be a lot better. Using techniques learned from the meta id generator, we can do some incredible stuff.Converting extensions to bundles will mean that we can have robust bundle dependencies, and in the future, do all Aquameta development within the bundle VCS, including core. It means everything will be data except for the bootstrap. We'll be free of the extension system's many shortcomings. It means future updates to Aquameta will just be another bundle pull.
General approach:
extension_to_bundle(extension_name, bundle_name)
:pg_catalog.pg_depends
which is where the contents of an extension is stored (by oid :/)meta
catalogOpen questions: