Should have delta generation & size estimate methods

thgreasi commented 5 years ago

trigger delta between two releases. This can be done using the API call:

https://delta.balena-cloud.com/api/v3/delta?src=<SRC_IMG>&dest=<DEST_IMG>
// for balenaOS < 2.47
https://delta.balena-cloud.com/api/v2/delta?src=<SRC_IMG>&dest=<DEST_IMG>

where is the source/pinned release image ID (or name) and is the “latest” image. The method could look like:

delta.generate(options: {
sourceRelease: string;
targetRelease: string;
version?: 'v2' | 'v3' // default to 'v3'
}): Promise<void>;

calculate the delta size between the images of the two releases
```
release1 = <OLD_RELEASE_ID>
release2 = <NEW_RELEASE_ID>
```

await sdk.pine.get({ resource: 'release', options: { $filter: { id:{ $in: [ release1, release2 ]}, }, $expand: { imageis_part_ofrelease: { $select: 'id', $expand: { image: { $select: 'id', $expand: { is_a_build_ofservice: { $select: 'service_name' } } } } } }, $orderby: 'id asc', } }).then(([r1, r2]) => { const r1services = { }; r1.imageis_part_ofrelease.forEach((ipr) => { r1services[ipr.image[0].is_a_build_ofservice[0].service_name] = ipr.image[0].id; }); const r2services = { }; r2.imageis_part_ofrelease.forEach((ipr) => { r2services[ipr.image[0].is_a_build_of__service[0].service_name] = ipr.image[0].id; });

const deltaSizes = { };
return Promise.all(Object.entries(r1services).map(([name, id]) => {
    return sdk.pine.get({
        resource: 'delta',
        options: {
            $select: 'size',
            $filter: {
                originates_from__image: id,
                produces__image: r2services[name]
            },
        }
    }).then(([ img ]) => {
        if (img != null) {
            deltaSizes[name] = img.size;
        } else {
            deltaSizes[name] = 0;
        }
    });
})).then(() => console.log(deltaSizes))

})



See: https://www.flowdock.com/app/rulemotion/resin-frontend/threads/TqmtsKrTgWvEWJVYuw2XZ6bRQtM
See: https://jel.ly.fish/0709f645-ba4a-487e-980f-507e07ebdbf7
See: https://github.com/balena-io/balena/issues/865

balena-ci commented 5 years ago

[gelbal] This issue has attached support thread https://jel.ly.fish/#/support-thread~0709f645-ba4a-487e-980f-507e07ebdbf7

ebradbury commented 3 years ago

@thgreasi Has this endpoint been implemented? I'm looking for a way to compare delta sizes across updates

thgreasi commented 3 years ago

No @ebradbury this hasn't been implemented yet in the SDK.

ebradbury commented 3 years ago

@thgreasi Are release deltas available through some other endpoint?

thgreasi commented 3 years ago

@ebradbury I think I misunderstood you. Can you clarify which endpoint you mean? I originally answered about the SDK method not being implemented yet.

ebradbury commented 3 years ago

@thgreasi I guess I'm wondering if image delta sizes are available through either the API or the SDK. How are they generated/displayed when doing balena push <app-id>?

https://www.balena.io/docs/reference/api/overview/ https://www.balena.io/docs/reference/sdk/node-sdk/

thgreasi commented 3 years ago

@ebradbury when doing a balena push, the cloud builder is the one that initiates the aut-generation of the new delta based of the previous release & the release that most devices of the fleet are not. All other deltas (between any other two releases) can be generated on the fly as soon as a device asks for them.

connormcmk commented 3 years ago

But the question is how to generate them manually without having to push them to a device with the right version

El El vie, ene. 22, 2021 a la(s) 2:22 a. m., Thodoris Greasidis < notifications@github.com> escribió:

@ebradbury https://github.com/ebradbury when doing a balena push, the cloud builder is the one that initiates the aut-generation of the new delta based of the previous release & the release that most devices of the fleet are not. All other deltas (between any other two releases) can be generated on the fly as soon as a device asks for them.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/balena-io/balena-sdk/issues/747#issuecomment-765263166, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIKHA6N2RKHXKF36KUN5Z3S3E7VLANCNFSM4I43RCIA .

ebradbury commented 3 years ago

I'm actually just wondering if I can retrieve that calculated delta size after the build completes. I'd like to be able to programmatically check the size of my releases prior to deploying them.

thgreasi commented 3 years ago

@cnrmck there are two part in this issue, for which we should have two separate methods:

calculate the delta size between the images of the two releases
trigger the generation of a delta between two releases that do not atm have one

@ebradbury the method from point (1) should allow you to do that. The proper use case would be to have your application pinned an a specific release, then create a new release, and finally only switch the tracked release to a newer one iff the size of the generated delta is below a threshold.

Let me also add a reference of our docs & the respective SDK method for application release pinning See: https://www.balena.io/docs/learn/deploy/release-strategy/release-policy/#pin-application-to-a-release See: https://www.balena.io/docs/reference/sdk/node-sdk/#balena.models.application.pinToRelease

connormcmk commented 3 years ago

Yeah, I’m mostly interested in #2, I have a way to do #1

El El vie, ene. 22, 2021 a la(s) 10:51 a. m., Thodoris Greasidis < notifications@github.com> escribió:

@cnrmck https://github.com/cnrmck there are two part in this issue, for which we should have two separate methods:

calculate the delta size between the images of the two releases

trigger the generation of a delta between two releases that do not atm have one

@ebradbury https://github.com/ebradbury the method from point (1) should allow you to do that. The proper use case would be to have your application pinned an a specific release, then create a new release, and finally only switch the tracked release to a newer one iff the size of the generated delta is below a threshold.

Let me also add a reference of our docs & the respective SDK method for application release pinning See: https://www.balena.io/docs/learn/deploy/release-strategy/release-policy/#pin-application-to-a-release See: https://www.balena.io/docs/reference/sdk/node-sdk/#balena.models.application.pinToRelease

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/balena-io/balena-sdk/issues/747#issuecomment-765582409, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIKHA5COB5YSF4KF5WFDQLS3G3IJANCNFSM4I43RCIA .

thgreasi commented 3 years ago

@cnrmck I've passed this request internally and we are at the moment investigating the best way to implement this. Two points that we care about are:

have a simpler way to trigger them, by just passing the two releases for which you care to have all the image deltas generated
keeping this operation as part of the API model, so that the SDK doesn't have to reach to the delta server directly.

connormcmk commented 3 years ago

Yep, makes sense to me. Thank you!

El El lun, ene. 25, 2021 a la(s) 3:34 p. m., Thodoris Greasidis < notifications@github.com> escribió:

@cnrmck https://github.com/cnrmck I've passed this request internally and we are at the moment investigating the best way to implement this. Two points that we care about are:

have a simpler way to trigger them, by just passing the two releases for which you care to have all the image deltas generated

keeping this operation as part of the API model, so that the SDK doesn't have to reach to the delta server directly.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balena-io/balena-sdk/issues/747#issuecomment-767157488, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIKHA4SJH5UL2CVYCU7HN3S3XWV3ANCNFSM4I43RCIA .

jellyfish-bot commented 3 years ago

[thgreasi] This issue has attached support thread https://jel.ly.fish/92e9d913-f413-4af2-b1bf-03d2c3be87b5

dfunckt commented 3 years ago

We published an experimental command line tool and NodeJS library that can be used to trigger deltas and fetch information about the update between two releases. You can find it here: https://github.com/balena-io-modules/balena-release-update

Please let us know if this works for you or any feedback you may have. Our plan is to ultimately officially adopt it in the product and make it available via the SDK, CLI and Dashboard but your feedback can help shape its exact form. Thanks!

connormcmk commented 3 years ago

Yay!!

On Thu, Aug 5 2021 at 3:47 AM, dfunckt @.***> wrote:

We published an experimental command line tool and NodeJS library that can be used to trigger deltas and fetch information about the update between two releases. You can find it here: https://github.com/balena-io-modules/balena-release-update

Please let us know if this works for you or any feedback you may have. Our plan is to ultimately officially adopt it in the product and make it available via the SDK, CLI and Dashboard but your feedback can help shape its exact form. Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balena-io/balena-sdk/issues/747#issuecomment-893320848, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIKHA7MGBXY2OBDL65OE4TT3JM2PANCNFSM4I43RCIA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

balena-io / balena-sdk

Should have delta generation & size estimate methods #747