Clarify which Database supports the LoopBack auto upgrade-migrate feature

[fpanaget]

The page Creating a database schema from models https://loopback.io/doc/en/lb3/Creating-a-database-schema-from-models.html#auto-update lists only 4 data sources for the auto-migration support.

The following data sources support auto-migration: Oracle PostgreSQL MySQL SQL Server MongoDB

Please update this this to list ALL the data sources currently available that support this feature. There is nothing mentioned about the auto-update. Should one assumes that if the auto-migration is supported the auto-update is?

Example: The DashDB connector appears to be supporting auto-update but it is not listed as supporting it.

Alternatively, you can create and configure the data source in JavaScript code. For example:

var DataSource = require('loopback-datasource-juggler').DataSource;
var DASHDB = require('loopback-connector-dashdb');

var config = {
  username: process.env.DASHDB_USERNAME,
  password: process.env.DASHDB_PASSWORD,
  hostname: process.env.DASHDB_HOSTNAME,
  port: 50000,
  database: 'SQLDB',
};

var db = new DataSource(DASHDB, config);

var User = db.define('User', {
  name: { type: String },
  email: { type: String },
});

db.autoupdate('User', function(err) {
  if (err) {
    console.log(err);
    return;
  }

---

Updated by @dhmlau

Acceptance Criteria

• [ ] For connectors with "supportedByStrongLoop": true, update available-connectors.json to reflect the correct information.
• [ ] based on the info gathered in available-connectors.json, build a table of which connectors support migration & discovery.

[crandmck]

Thanks @fpanaget, good suggestion...

Should one assumes that if the auto-migration is supported the auto-update is?

I believe that is the case, but if so we should say that explicitly.

@smartmouse Do we have a complete list of connectors that support auto-migrate/update?
And is it true that if it supports auto-migration then it also supports auto-update?

[smartmouse]

I do not, @0candy do you know who work on the features and if we can obtain the information as requested?

[0candy]

This is the list where I would start with. It list the connectors and whether it supports migration. https://github.com/strongloop/loopback-workspace/blob/master/available-connectors.json

However, the ones that have false may not mean they don't work. Like in the case above with dashdb, could just be that we haven't thoroughly tested it so we don't list it as supported yet.

It would make sense that if it supports auto-migration then it also supports auto-update. Without confirming the code in each connector, I cannot be sure.

@raymondfeng would probably be a better person to confirm with.

[crandmck]

OK, I think we should have a "supported" list that comes from https://github.com/strongloop/loopback-workspace/blob/master/available-connectors.json.

We can also have a list of "should work, but not tested/supported--use at your own risk" - Who can provide me with this list?

From available-connectors.json for "migration": true, I get the following:

• MongoDB
• MySQL
• PostgreSQL
• Oracle
• MS SQL Server
• SAP HANA (Community connector) ... Not sure we should say this is "supported"?

[smartmouse]

@0candy @raymondfeng ^^ : I like @crandmck 's suggestion.
"We haven't thoroughly tested them" => but need more information so that we can come up with what @crandmck needs and then improve upon them as we fully tested all of them.

For -auto-migrate and auto-update features, which cases are these for each of the connectors?

1. We have testcases in CI but need to improve on coverages.
2. We do not have testCases at all and need to create them.
3. We have testcases created and run in CI but failed and we have not done our investigation yet.
4. The features are not fully implemented yet and WIP.
5. Others -- specify please

[smartmouse]

From the information that I obtained with @jannyHou 's help

All non-sql datasources - 5. The features will not be implemented and are not supported. mongodb etc..

db2 connector - 3. The features should be supported but not currently working with apic.

db2 for zOS -- 5. The features are not supported at this point.

mysql/mssql/oracle/postgresql support both features.

[crandmck]

All non-sql datasources - 5. The features will not be implemented and are not supported. mongodb etc..

That doesn't square with https://github.com/strongloop/loopback-workspace/blob/master/available-connectors.json#L242 which indicates that MongoDB connector should support it, right?

We currently document that Mongo supports it, so please let me know if I should change that.

db2 connector - 3. The features should be supported but not currently working with apic.

Does that mean it works outside APIC, i.e. if you just run the app using node .? If so, should I document it?

[jannyHou]

Sorry I didn't find the migration.js in noSql connectors and didn't realize that they are implemented in the main file.

We support autoupdate and automigrate in mongodb. Mongodb will do an auto-migration when datasource connected, see https://github.com/strongloop/loopback-connector-mongodb/blob/master/lib/mongodb.js#L1384

And I also notice that cloudant actually has same implemenation, but I need to double check is it working...especially with apic.

[smartmouse]

you are right, cloudant has automigrate and discovery.js too https://github.com/strongloop/loopback-connector-cloudant/blob/master/lib/cloudant.js#L813

[crandmck]

OK, so the upshot so far is:

• Cloudant supports automigrate (and discovery):
  • Should I add this to docs?
  • Has it been tested?
  • Why isn't this reflected in available-connectors.json ?
• DB2 automigrate is "... supported but not currently working with apic"
  • Should I change LB docs to say the DB2 connector supports update/migrate?
  • In general, we haven't documented differences between using connectors with open-source LoopBack vs. with APIC. Are there differences? How can that be? Should I document them?

[smartmouse]

@jannyHou Even mongodb does not make sense to me. You mention the two features depend on each other and need to work together, mongodb has true/false in the available-connectors.json

[smartmouse]

@crandmck : for DB2, @jannyHou can probably explain better than me, the issue with apic side of things are the Toolkit does not return the output from the connector properly, that is why it does not work with apic. There is an issue opened for that already. So I would say if the db2 connector supports the two features and works fine, we can document it as supported. For cloudant, will defer to @jannyHou to comment.

How urgent is this documentation? We may have more information during the LB3 testing that will be in this sprint.

[crandmck]

How urgent is this documentation?

IMO, not urgent, since the doc as been "as is" for some time now. But of course, I always want to make the doc as accurate as possible. :-) Waiting for the LB3 testing in next sprint is probably fine.

[fpanaget]

Waiting for the LB3 testing in next sprint is probably fine.

That seems reasonable.

[crandmck]

@smartmouse Just one other thing re your comment:

[...] mongodb does not make sense to me. You mention the two features depend on each other and need to work together, mongodb has true/false in the available-connectors.json

I think @0candy's comment was that auto-update and auto-migration are closely related/linked; not that auto-update/migrate are linked to discovery. The docs for implementing auto-migrate/update and implementing discovery imply this.

SO IIUC, it's entirely possible from technical perspective that a given connector could support auto-migrate/update and discovery independently (i.e. might support one but not the other).

But, at a higher level, a user might expect both features to be implemented, and if one is not that would be considered a "missing feature". :-)

[jannyHou]

@smartmouse This is the supported functions in mongodb and cloudant:

	automigrate	autoupdate	discoverModelDefinitions	discoverSchemas
loopback-connector-mongodb	Y	Y	N	N
loopback-connector-cloudant	Y	Y	Y	Y

Sorry my previous explanation make you confused. Let me elaborate more. By "migration relies on discovery" I mean for some connectors, e.g. loopback-connector-mysql some migration function calls functions in discovery, example see autoupdate calls discoverForeignKeys

While in mongodb and cloudant, they are totally independent ones at this moment.

Back to mysql, the example I post is from a recent community pr, and FK migration/udpate haven't been introduced to other connectors, so I believe for most connectors, the two features are still independent.

[crandmck]

OK, I see.

As an aside, I don't think MongoDB would ever support discovery since it's not an RDB. Instead it supports creating models by "instance introspection" instead of discovery. See http://loopback.io/doc/en/lb3/Creating-models-from-unstructured-data.html.

[dhmlau]

There seems to be 2 problems:

1. the docs does not have a complete list of which connectors support automigrate or discovery or both. https://loopback.io/doc/en/lb3/Creating-a-database-schema-from-models.html#auto-update
2. the available-connectors.json does not provide the correct information, and thus cannot be the place of finding out the truth.

Acceptance Criteria

• [ ] For connectors with "supportedByStrongLoop": true, update available-connectors.json to reflect the correct information.
• [ ] based on the info gathered in available-connectors.json, build a table of which connectors support migration & discovery.

Will update the original description to contain the acceptance criteria.

[dhmlau]

Fixed in https://github.com/strongloop/loopback.io/pull/702. Closing this issue.