DM/AS Connection (Stage 1)

[alanbchristie]

The Data Manager (DM) will soon be able to communicate with the Account Server (AS).

In Stage 1 only Storage statistics will be delivered, which will be achieved via a "private" channel and won't need the UI.

UI changes (A) - Assisting the DM to connect to the AS

What the UI will need to do to facilitate this is to provide "real" Organisation, Unit and Product identities when using the following DM-API endpoints: -

• POST /project
• POST /dataset
• PUT /dataset

The AS can serve-up these identities using: -

• GET /organisation
• GET /organisation/{org_id}/unit
• GET /product/organisation/{org_id}/unit/{unit_id}

But you can use constants, as they're built-in and will not change for this stage: -

ORG_ID: str = 'org-00000000-dead-0000-0000-000000000000'
UNIT_ID: str = 'unit-00000000-dead-0000-0000-000000000000'
PRODUCT_DATASET_ID: str = 'product-00000000-dead-0000-0000-00000000000d'
PRODUCT_TIER_EVALUATION_ID: str = 'product-00000000-dead-0000-0000-00000000000e'
PRODUCT_TIER_BRONZE_ID: str = 'product-00000000-dead-0000-0000-00000000000b'
PRODUCT_TIER_SILVER_ID: str = 'product-00000000-dead-0000-0000-00000000000s'
PRODUCT_TIER_GOLD_ID: str = 'product-00000000-dead-0000-0000-000000000009'

Getting the IDs from the AS could be implemented but, to minimise development effort, the UI could use the above constants for the time-being.

Note - the AS still does not use authentication and all users belong to all Units and Organisations.

UI changes (B) - Getting storage statistics

Once the UI starts to pass orgs, units and products into the DM-API (development in progress) statistics will be available on the AS endpoint: -

• GET /product/organisation/{org_id}/unit/{unit_id}

At the moment only Storage will be recorded but (when the DM to API server connection is complete) you should start to see "real" storage values and coin usage using the above product endpoint. You should see storage size, coins, burn rates and spend predictions.

[alanbchristie]

The AS now keeps a track of the total size of all newly uploaded Datasets.

No Project or Project Storage measurements are handled atm. So the four Project Tier products will not change. Only the DATA_MANAGER_STORAGE_SUBSCRIPTION changes.

To exercise the DATA_MANAGER_STORAGE_SUBSCRIPTION you need to use the hard-coded organisation_id and unit_id values on the DM-API’s PUT /dataset endpoint. When the Dataset has finished uploading (which may take seconds or minutes depends on the size of the file your file) the DM sends a “Storage” message to the AS on behalf of the Unit so that it can keep track of the Unit’s current and peak storage values.

When a Dataset is removed another “Storage” message is sent to the AS so that it can adjust the current storage measurement for the Unit.

When you do a GET on the AS /product/organisation/{org_id}/unit/{unit_id} endpoint you will see changes to the DATA_MANAGER_STORAGE_SUBSCRIPTION object.

Specifically: -

• Storage is charged at 0.1 coins per MiB per day (or part thereof) - i.e. 1 byte costs 0.1 coins
• storage.coins.used will have a value based on storage.size.used
• storage.size.used is a “humanised” PEAK value for the day (i.e. “158.4 KiB”)
• storage.size.current is the Unit’s “humanised” current consumption
• coins.billing_prediction is the predicted coin-cost of the subscription when the billing period ends (in 14 days for this experimental system)
• coins.used is what the Unit has committed to spend so far
• coins.current_burn_rate is the coin value being used to maintain your current Dataset files

storage.size.current is a new property in the response as storage.size.used is now used for the "peak" value.

If you look now you’ll see 158.8 KiB used (peak) with a current value of 0 Bytes. That’s because I uploaded two small datasets and then deleted them. That cost 0.1 coins with a billing prediction of 0.1. That’s correct because my “peak” was 0.1 coins but as I’m not using Dataset storage any more that’s my total bill. If we upload new datasets the prediction should reflect the new burn rate.