admin/bump-cdp-backend-3.0.0.dev19

[isaacna]

Description of Changes

• Bumping cdp-backend to 3.0.0.dev19

[evamaxfield]

Would love your opinion on this one.

I was waiting for test-deployment to run with the latest cdp-backend to make this dep bump. It did and completed just fine. (check the python dep install log to see that it installed v3.0.0.dev19)

Then, because I already added a quickstart Python to the README I checked the following:

from cdp_backend.database import models as db_models
import fireo
from google.auth.credentials import AnonymousCredentials
from google.cloud.firestore import Client

# Connect to the database
fireo.connection(client=Client(
    project="cdp-test-deployment-435b5309",
    credentials=AnonymousCredentials()
))

# Get all transcripts from db
transcript_models = list(db_models.Transcript.collection.fetch(1000))

# Get a list of generators used
generators = []
for t in transcript_models:
    if t.generator not in generators:
        generators.append(t.generator)

And generators results in:

[None,
 'CDP WebVTT Conversion -- CDP v3.0.0.dev19',
 'Google Speech-to-Text -- CDP v3.0.0.dev19']

Basically, all prior documents that are missing the field store None. Works for me for now. But this is likely good documentation for us in the future. When we roll out database changes we should maybe ship a script that runs alongside the infrastructure build? Like cdp-stack builds itself, and then a script runs after the stack updates to make any additional changes?

Anyway. Feel free to merge whenever, just would love your thoughts on this.

[isaacna]

Cool, glad to see the test deployment works!

When we roll out database changes we should maybe ship a script that runs alongside the infrastructure build? Like cdp-stack builds itself, and then a script runs after the stack updates to make any additional changes?

What exactly do you mean by "additional changes?" Do you mean like a backfill script that for the deployment, updates its old db items to match the new model? If so, maybe we just that as part of the CDPStack creation after this. Or do you mean like a script that actually updates the package requirement changes when there's a backend dev bump? I'm not too sure what changes you're referring to

And I'll merge this change in, but we can continue tracking this topic here!

[evamaxfield]

Yea sorta kinda. We can't really put any script stuff in the CDPStack object file / object. That's just not how pulumi works unfortunately. But what I think what we should do is have a script that is called like cdp-post-infra-build-upgrades that runs a bunch of functions that apply some changes / fixes where needed. Then the cookie cutter to the build infra workflow we run pulumi then run that script.

So like cdp_backend/infra/updates.py is a file that has update_3_0_1 and update_3_0_2 etc. Then we have a script that runs all the functions for 3_0_x functions.

Idk if that makes sense. We can discuss on Tuesday. It's similar to a great system called alembic

[isaacna]

Ah ok that makes more sense. I'm not too clear on what types of updates these would be but the deployment process makes sense. And yeah I'm down to discuss next meeting!