First step to issue #359, no change to the HTML output

[finanalyst]

• add sqlite-db plugin
• plugin accesses the dataset created for the Routines page in the website
• the data is formatted into the rows of an sqlite table
• the completed sqlite database is output into a filename specified in the configs/03-plugin-options.raku config file, so changing the value of the db-filename field of the sqlite-db sub-hash changes the output file name.
• the sqlite file is moved by Collection to a directory defined by 'database-dir' relative to the directory in which Collection runs. As set up here, the directory sqlite_dir will be at the same level as the existing renedered_html

[finanalyst]

@dontlaugh may be this branch could be made into something like the docs-dev branch? Then you could make commits to it to access the sqlite database

[finanalyst]

@patrickbkr @dontlaugh the sqlite file created when this plugin is run during a build of the website is a valid database file. I tested it by installing sqlite3, opening it with no database, then running .read sqlite-db-definition.sql. A select * from routines generates all the data. limit the output as there are hundreds of rows.

[dontlaugh]

@dontlaugh may be this branch could be made into something like the docs-dev branch? Then you could make commits to it to access the sqlite database

No. I don't want to have another custom deployment branch.

[finanalyst]

@dontlaugh Attached is a zip of the sqlite file generated from a test collection of RakuDoc documents. I'm not sure what you mean by 'on the fly'. The PR here is a plugin that uses data collected during a build of the HTML site and formats it into a sql file. A similar dataset that can be reformatted is the data used for the search plugin. Is this what you mean by 'on the fly'?

Two things:

• Instead of creating a simple table with a dataset created for the website, the same plugin technique could be used to generate an sqlite file with all the accumulated data (see the specification of PodFile in Raku::Pod::Render), which is passed in the %processed parameter provided to the compilation stage plugin. However, this is again 'on the fly' (if I understood you properly).
• It would be possible to separate out the generation of a sqlite db from the Build of the website, by creating another mode (similar to Website and EBook), eg. SQL_database.
  • an advantage of this is that SQL_database could be run independently of Website.
  • the templates for each page could be changed without affecting the HTML templates for the website.
  • the disadvantage is that all the RakuDoc sources will need to be rendered separately, eg, once each for each mode that is run

sqlite.zip

[dontlaugh]

What I mean is that I would prefer it if this PR contained one standalone SQL file with the following contents:

CREATE TABLE IF NOT EXISTS routines (
    Category TEXT,
    Name TEXT,
    Type TEXT,
    URL TEXT
);

-- future schema changes below...

This lets us test updates to database structure without invoking a build. I understand that this would yield an empty database. That's actually good, because schema changes will often necessitate changes to INSERT statements.

Could such a schema.sql could live next to the other plugin code?

[finanalyst]

@dontlaugh I am trying to understand this request. Are you saying that there should be two files, a schema file like the one above, and a data one with the INSERT ... lines? The build process would only generate the data file.

So we could have

• The schema file is stored in a sub-directory in the plugin, and just copied to the sqlite-dir.
• The build generates a Data.sql file and transfers that to the sqlite-dir.
• We could then tweak the schema file, then read in the data. Once a schema is settled on, a separate PR could change the schema in the plugin.

I'll add commits to the branch to do this.