search

geosolutions-it / C195-azure-workspace

1 stars 2 forks source link

deployment automation of ckan as a vm with docker #6

Closed lpasquali closed 3 years ago

lpasquali commented 3 years ago

provisioning of ckan vm is automatic but lacks of installation of needed software and config files for ckan

lpasquali commented 3 years ago

The generated json for whole resource group required some more parametrization and modifications to be run elsewhere. I almost finished in deploying a complete resource group with custom parameters (for hostnames, resources, so on).

lpasquali commented 3 years ago
lpasquali commented 3 years ago

I made a pull request for my automatic approach to deploy everything unattended here https://github.com/geosolutions-it/C195-azure-workspace/issues/4

lpasquali commented 3 years ago

started working to fix deployment issues:

lpasquali commented 3 years ago

@etj I was able to reproduce the issue we had yesterday with init db as well on the environment where we have the running ckan (resource group CREA_test_20210120), the very same sql alchemy url behaves differently when application does db init (that is a command done at application yet not running) than when application starts, in this latter case the sqlalchemy url is good and the '@' escaping we worked out works but in the init db phase:

invalid interpolation syntax in 'postgresql://ckan%40creapostgresqltest :XXXXXXXXXXXXXXX@crea-pg.privatelink.postgres.database.azure.com/ckan' at position 17

I was able to make it work patching ckan like this:

    def setup_migration_version_control(self):
        self.reset_alembic_output()
        alembic_config = AlembicConfig(self._alembic_ini)
        dburl = str(self.metadata.bind.url).replace('%', '%%')
        alembic_config.set_main_option(
            "sqlalchemy.url", str(dburl)
        )

        try:
            sqlalchemy_migrate_version = self.metadata.bind.execute(
                u'select version from migrate_version'
            ).scalar()
        except ProgrammingError:
            sqlalchemy_migrate_version = 0

        # this value is used for graceful upgrade from
        # sqlalchemy-migrate to alembic
        alembic_config.set_main_option(
            "sqlalchemy_migrate_version", str(sqlalchemy_migrate_version)
        )
        # This is an interceptor for alembic output. Otherwise,
        # everything will be printed to stdout
        alembic_config.print_stdout = self.add_alembic_output

        self.alembic_config = alembic_config

I committed in this branch here

lpasquali commented 3 years ago

All fixes should be in place, my only problem to release for the client is that I changed submodule in the branch where I placed the fixes to use my ckan patch, anyway, it seems I can updated the submodule once the git url changed, so the script which configures git repository in azure for the ckan docker image is not working and requires manual intervention because the database is not initalized correctly.

lpasquali commented 3 years ago

Deployment was tested from scratch a few times. For "deployment" I mean whole resources are up, I can login with a default password as admin in ckan, I can run the script of #12 and set 3 orgs and 3 datasets as initial data, human interactions are:

I have all needed modifications were pushed to branch envoronment-fixes on geosolutions-it/C195-azure-workspace repository. I needed to patch ckan init module in models here in order to populate ckan database with initial data. In order to proceed to merge everything and have a master branch with all needed modifications and pointing to the correct submodule I'd need help from you @etj and a revision of my work as well.

etj commented 3 years ago

For the CKAN issue I opened an issue on the official CKAN repo and proposed a PR:

lpasquali commented 3 years ago

@etj I made a PR for every needed fix here: https://github.com/geosolutions-it/C195-azure-workspace/pull/18

lpasquali commented 3 years ago

@etj can we close this?