CartoDB / carto-python

CARTO Python client
https://carto.com
BSD 3-Clause "New" or "Revised" License
154 stars 62 forks source link

Not possible to create syncing DB connection #145

Open gelin opened 5 years ago

gelin commented 5 years ago

The syncing DB connections, as described in Import API are POSTs with JSON content to "/api/v1/synchronizations" where both connector and interval are defined. Also, actually there are null data import urls (archive) there.

The code in datasets.py does not allow to do this:

    def is_sync_table(self, archive, interval, **import_args):
        return (hasattr(archive, "startswith") and archive.startswith("http")
                or "connection" in import_args)    # this is true for DB connections
            and interval is not None               # this also is true for syncing connections

    def create(self, archive, interval=None, **import_args):
        # ...
        if self.is_sync_table(archive, interval, **import_args):  # true for syncing DB connection
            manager = SyncTableJobManager(self.client)
        else:
            manager = FileImportJobManager(self.client)

        if interval is None:
            import_job = manager.create(archive)  # different signature for different managers (out of this issue)
            import_job.run(**import_args)
        else:
            import_job = manager.create(archive, interval)  # this is called for SyncTableJobManager
            # !!! import_args (including "connection") are just ignored and not POSTed later
        # ...
gelin commented 5 years ago

Also, it's even not possible to send necessary JSON to Carto API for this case.

It's necessary to POST JSON like:

{
  "connector": {
    "provider": "postgres",
    "connection": {
      "server": "postgres.server",
      "database": "db",
      "username": "dbuser",
      "password": "dbpass"
    },
    "schema": "dbschema",
    "table": "dbtable"
  },
  "interval": 86400
}

However, SyncTableJobManager does this:

resource = self.resource_class(url, interval, self.client)
resource.update_from_dict(kwargs)
resource.save(force_create=True)

It tries to create a resource of type SyncTableJob which has no "connector" property. So connector parameters are never sent to the server.