obmd migration strategy

[zenhack]

A spec has been merged describing the changes needed to HIL to integrate with OBMd. One question still floating around in my mind is how we're going to go about migrating existing deployments; Doing this naively would require removing all nodes and re-adding them. It would be nice to find a less disruptive way to deal with this.

Open to input, and in particular insight into how much disruption is acceptable.

[naved001]

I guess we would get rid of all obm information from the node table (so a migration script for that) leaving only node label and a pointer to it's obmd counterpart, and then another script to add a table for obmd (another migration script for that). The latter might be different if we are going to have a different database altogether. It would be clearer if we first decide how the database is going to look like.

how much disruption is acceptable.

AFAIK, hil is only deployed on our own clusters that I am managing, so at least I would be fine with some disruption.

@SahilTikale since you are back, would appreciate your thoughts on the discussions here. @razaaliraza @mosayyebzadeh feel free to contribute to the discussion on the threads here.

[zenhack]

We could write a couple helper scripts specifically for this use case:

• One that extracts the node info from the HIL database, and creates the nodes in OBMd.
• One that adds the obmd connection info to the nodes, and possibly deletes the ipmi connection info.

We could also do the database migration in two steps:

1. Add the obm uri & admin token fields to the node model. For now, allow them to be null and don't use them for anything. This is a small patch that can go in now without any other work.
2. The next migration script will come with release notes that notify the adminstrator that they must (1) upgrade to the previous version first (i.e. they can't "skip" it), and (2) run the above scripts to set up the data. Then, the migration script sets the obmd connection info fields as not-nullable, and deletes the obm/ipmi tables.

The current design assumes obmd has a separate database.

[zenhack]

@naved001, thoughts on the above?

[naved001]

okay, so the order in which somebody would run these is:

1. Use helper-script (1) to extract the info from HIL DB and create nodes in OBMd.
2. Then migration-script (1) will add the obm uri and admin token fields, and allow those to be null.
3. Then helper-script(2) will feed the OBMd connection info into the HIL DB. How will this one work? I am imagining updating the rows in the DB directly. (because we can't remove nodes, and re-add them; they might not be free).
4. The final migration-script(2) will delete the old obm tables/columns, and make the OBMd stuff not nullable.

Is that correct?

[zenhack]

Quoting Naved Ansari (2018-01-09 17:39:04)

okay, so the order in which somebody would run these is:

1. Use helper-script (1) to extract the info from HIL DB and create nodes in OBMd.
2. Then migration-script (1) will add the obm uri and admin token fields, and allow those to be null.
3. Then helper-script(2) will feed the OBMd connection info into the HIL DB. How will this one work? I am imagining updating the rows in the DB directly. (because we can't remove nodes, and re-add them; they might not be free).

4. The final migration-script(2) will delete the old obm tables/columns, and make the OBMd stuff not nullable.

   Is that correct?

Yeah, that's what I was thinking.

[naved001]

This plan sounds good to me then. Just one thing though, if we do migration-script(1) first, then we could combine the two helper scripts, though it could make it more complex.

[zenhack]

Yeah, the reason to keep them separate is just so we don't have to have the migration script be "different" from the other migration scripts.

[zenhack]

But yeah, we could combine 1 & 3.

I'll get started.

[naved001]

Are you working on this? I can pick this up if you are doing other things.

[mosayyebzadeh]

Guys i am getting emails about this issue. Am I assigned to this issue? If yes, I did not know that.

[naved001]

you aren't assigned. You could unsubscribe from this issue.

[zenhack]

@naved001, I've got some work on the scripts on a branch, but have distracted by other things. I'll try to find some time next week.

[zenhack]

Do we want to keep this open until this stuff has actually landed? it seems like we've come to a consensus on the actual thing.

[zenhack]

All of this stuff has landed; I'm going to close.