Why is config in database rather than files?

[kelizoliva]

We noticed that most of the config is being stored in the database as json. We were wondering why configuration was not instead writing to the config file? Is this something that is in progress somewhere that we are just not seeing?

[stpaultim]

@kelizoliva - Can you provide a little more information about your question/concern?

It is my understanding that most configuration for Backdrop CMS core is stored in JSON files in the config directory. I am not aware of it being stored in the database at all and I'm not sure what you mean by storing it in the database as json? (But, there may be things going on that I am unaware of or don't fully understand).

I have heard that some contributed modules may still store their config in the database, but in that case I would not expect it to be in JSON, but I may be wrong about that.

So when you say that "most of the config is being stored in the database as json," can you say more about what you mean. Core config? Config for contrib modules? Both?

I know that there items that are stored in the database that many could consider config and this might be what you are referring to, which is why specific examples might help.

For example menu items and/or taxonomy terms might be considered by some to be config and other consider content.

[kelizoliva]

@stpaultim I was speaking specifically to this module's configuration, not backdrop configuration generally; I'm not sure if something I said was misleading and sounded like I was referring all Backdrop configuration, but that was not my intention. My question is specific to the backup and migrate module configuration, the module represented in this particular code repository that I have opened this issue in.

So when you say that "most of the config is being stored in the database as json," can you say more about what you mean. Core config? Config for contrib modules? Both?

I am speaking specifically about this module. There is a backup_migrate.settings.json file in the config directory that stores some config info. Here is a sample of that file:

{
    "_config_name": "backup_migrate.settings",
    "backup_migrate_data_rows_per_line": 30,
    "backup_migrate_data_bytes_per_line": 2000,
    "backup_migrate_max_email_size": 20971520,
    "backup_migrate_allow_backup_to_file": true,
    "file_private_path": false,
    "file_public_path": false,
    "backup_migrate_allow_backup_to_download": true,
    "nodesquirrel_activate_url": "http://manage.nodesquirrel.com/activate",
    "nodesquirrel_manage_url": "https://manage.nodesquirrel.com",
    "nodesquirrel_schedule": 86400,
    "nodesquirrel_schedule_enabled": true,
    "nodesquirrel_schedule_source_id": "db",
    "nodesquirrel_secret_key": "",
    "nodesquirrel_endpoint_urls": [],
    "nodesquirrel_default_endpoint_urls": [
        "api.nodesquirrel.com/services/xmlrpc"
    ],
    "debug_http_request": false,
    "dev_query": 0,
    "site_offline_message": "backdropdev is currently under maintenance. We should be back shortly. Thank you for your patience.",
    "maintenance_mode": 0,
    "maintenance_mode_message": null,
    "backup_migrate_cleanup_time": 21600,
    "backup_migrate_cleanup_temp_files": true,
    "backup_migrate_profile_id": "civicrm_daily",
    "backup_migrate_smart_keep_hourly": 24,
    "backup_migrate_smart_keep_daily": 30,
    "backup_migrate_smart_keep_weekly": 2147483647,
    "backup_migrate_schedule_buffer": 0.01,
    "backup_migrate_default_schedule": 3600,
    "backup_migrate_source_id": "civicrm_database",
    "backup_migrate_destination_id": "download",
    "backup_migrate_copy_destination_id": "",
    "backup_migrate_backup_max_time": 1200,
    "backup_migrate_restore_max_time": 1200,
    "backup_migrate_timeout_buffer": 5,
    "nodesquirrel_schedule_keep": -2
}

In the database, there are 4 tables:

• backup_migrate_destinations
• backup_migrate_profiles
• backup_migrate_schedules
• backup_migrate_sources

Destination settings, profile filters, and sources settings are all stored in the database tables as JSON. More broadly, it strikes us as odd that any of the backup and migrate configuration is stored in the database, as this is not content. But it is strange that it is partially stored in a JSON file, and partially stored in the database. It seems that the defaults for backup_migrate are stored in the config file, and all additional destinations, profiles, schedules and sources are stored in the database. This seems really odd.

Why is so much of this config being stored in the database as JSON? Why is all of the configuration for this module not stored in the backup_migrate JSON file in the active config directory? Is there any effort that is not currently documented in this repo to shift that?

[quackers8me]

This may have been an oversight on my part when I initially ported it to Backdrop.

I believed that the workflow was to have the defaults load to the database and then the Backdrop export would be used to save the configuration to the filesystem if the server needed to be moved into production.

I can look at how the config workflow should be to match other module examples.

On Thu, Jan 7, 2021 at 5:32 PM kelizoliva notifications@github.com wrote:

@stpaultim https://github.com/stpaultim I was speaking specifically to this module's configuration, not backdrop configuration generally; I'm not sure if something I said was misleading and sounded like I was referring all Backdrop configuration, but that was not my intention. My question is specific to the backup and migrate module configuration, the module represented in this particular code repository that I have opened this issue in.

So when you say that "most of the config is being stored in the database as json," can you say more about what you mean. Core config? Config for contrib modules? Both?

I am speaking specifically about this module. There is a backup_migrate.settings.json file in the config directory that stores some config info. Here is a sample of that file:

{ "_config_name": "backup_migrate.settings", "backup_migrate_data_rows_per_line": 30, "backup_migrate_data_bytes_per_line": 2000, "backup_migrate_max_email_size": 20971520, "backup_migrate_allow_backup_to_file": true, "file_private_path": false, "file_public_path": false, "backup_migrate_allow_backup_to_download": true, "nodesquirrel_activate_url": "http://manage.nodesquirrel.com/activate", "nodesquirrel_manage_url": "https://manage.nodesquirrel.com", "nodesquirrel_schedule": 86400, "nodesquirrel_schedule_enabled": true, "nodesquirrel_schedule_source_id": "db", "nodesquirrel_secret_key": "", "nodesquirrel_endpoint_urls": [], "nodesquirrel_default_endpoint_urls": [ "api.nodesquirrel.com/services/xmlrpc" ], "debug_http_request": false, "dev_query": 0, "site_offline_message": "backdropdev is currently under maintenance. We should be back shortly. Thank you for your patience.", "maintenance_mode": 0, "maintenance_mode_message": null, "backup_migrate_cleanup_time": 21600, "backup_migrate_cleanup_temp_files": true, "backup_migrate_profile_id": "civicrm_daily", "backup_migrate_smart_keep_hourly": 24, "backup_migrate_smart_keep_daily": 30, "backup_migrate_smart_keep_weekly": 2147483647, "backup_migrate_schedule_buffer": 0.01, "backup_migrate_default_schedule": 3600, "backup_migrate_source_id": "civicrm_database", "backup_migrate_destination_id": "download", "backup_migrate_copy_destination_id": "", "backup_migrate_backup_max_time": 1200, "backup_migrate_restore_max_time": 1200, "backup_migrate_timeout_buffer": 5, "nodesquirrel_schedule_keep": -2 }

In the database, there are 4 tables:

• backup_migrate_destinations
• backup_migrate_profiles
• backup_migrate_schedules
• backup_migrate_sources

Destination settings, profile filters, and sources settings are all stored in the database tables as JSON. More broadly, it strikes us as odd that any of the backup and migrate configuration is stored in the database, as this is not content. But it is strange that it is partially stored in a JSON file, and partially stored in the database. It seems that the defaults for backup_migrate are stored in the config file, and all additional destinations, profiles, schedules and sources are stored in the database. This seems really odd.

Why is so much of this config being stored in the database as JSON? Why is all of the configuration for this module not stored in the backup_migrate JSON file in the active config directory? Is there any effort that is not currently documented in this repo to shift that?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/backdrop-contrib/backup_migrate/issues/60#issuecomment-756490509, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACQU5TZWNORWMYTHVB234J3SYZOBRANCNFSM4VZOFBDA .

[kelizoliva]

Just noting that the Backdrop config export output is identical to what is already in the config file, and is not exporting any of the additional config. So currently config is not portable/replicable.

[herbdool]

In my experience in converting modules from Drupal 7 to Backdrop, the easier part is converting settings in the variable table to JSON config files. The harder part is, first, deciding if other tables should also be in configuration files, and, second, recreating the UI for that part to work with config files.

[quackers8me]

@herbdool, Do you have an example of a module that you could suggest that has done this well?

[herbdool]

Off the top of my head I'd say Feeds, because I've helped with it.

[argiepiano]

Settings such as the ones found here: admin/config/system/backup_migrate/settings/profile/edit/default are currently saved as config json files - those were variables in D7. The issue, as pointed out above, is that destinations, sources, profiles and schedules are still saved in the db, which makes them unavailable to Backdrop config exports. Those can still be exported/imported through the Backup and Migrate UI.

BTW, this type of issue affects other modules like Webforms. It takes some delicate work to switch those db-stored instances of "stuff" into json config files - and the fact that there are almost 1000 installations of this modules, makes it even more tricky.

[yorkshire-pudding]

BTW, this type of issue affects other modules like Webforms. It takes some delicate work to switch those db-stored instances of "stuff" into json config files - and the fact that there are almost 1000 installations of this modules, makes it even more tricky.

Just to mention, Webform Share does this for Webform; it exports to json and imports from json; not as part of the main config (there are a few discussions as to why that is challenging) but as a standalone process.

[irinaz]

@kelizoliva , are you still working with config? Was original problem solved?