Remove calculation of storage ratios from default step data

[breathnach]

Currently the storage ratios for all storage agent_types are always calculated per step. These ratios are returned in the step data in the model_stats dictionary.

The change:

• remove these from default step data.
• calculate them on demand via a route.
• The format for saving the agent_types on the database is also changing. Instead of a list (or more recently a dictionary) of currencies for each storage agent_type, each {agent_type, agent_id,currencies name} combo will be stored/accessed separately

Implementation options: 1/ edit get_step to take as input a json with the step_num and a list of key:value pairs for agent_type:agent_id for which to calculate the ratios. I will then calculate the ratios for all listed, and return them in the same format as previously as part of the step_data. 2/ Have a separate route, where you pass a list of agent_type:agent_id and it returns the ratios 3/ One of the above, but specify agent_type:agent_id:currency_name, and only return the ratio for that value

You can see an example of the current step data json, including all this information, in: https://github.com/kstaats/simoc/issues/50#issuecomment-492230492

Side note: storage ratios for _ are the ratios of the currencies values.

[iurii-milovanov]

@breathnach please check our my recent changes to the abm_database branch (https://github.com/kstaats/simoc/commit/45f91b03483ac0a97e3be9df84bd0aad3c0d8fcf).

Note that storage statistics are now stored in the storage_capacity_record table and the agent counters are stored in the agent_type_count_record (instead of model_record).

Here are a couple of examples of how you can pull these tables:

Get all alive agents at the step 10:

from simoc_server.database.db_model import ModelRecord

step_data = ModelRecord.query \
            .filter_by(step=10) \
            .first()
values = [(i.agent_type.name, i.agent_counter) for i in step_data.agent_type_counters]

Get air_storage capacities at the step 10:

from simoc_server.database.db_model import ModelRecord, AgentType, StorageCapacityRecord

agent_type = AgentType.query \
            .filter_by(name='air_storage') \
            .first()

step_data = ModelRecord.query \
            .filter_by(step=10) \
            .first()

capacities = StorageCapacityRecord.query \
            .filter_by(agent_type=agent_type) \
            .filter_by(model_record=step_data) \
            .all()
values = [(i.currency, i.value) for i in capacities]

[iurii-milovanov]

Also note that in my recent commits (https://github.com/kstaats/simoc/commit/0e220ede597f703cf2279c09224f091657bc8fa5, https://github.com/kstaats/simoc/commit/638252da4351dd5921ef94682296e01502e3b72b & https://github.com/kstaats/simoc/commit/9feb32b7ac3b436522a474638eccb0b2f3303b06) to the abm_database branch I changed to logic of the GameRunnerManager.get_step() method and introduced new GameRunnerManager.get_steps() and GameRunnerManager.get_last_steps() methods. They all now do not perform any computations, but only pull the data out of the database.

You should use the GameRunnerManager.get_step_to(user, step_num, buffer_size) method instead to request the ABM to run the simulation up to the step_num step. It will continuously populate the DB every buffer_size steps.

[iurii-milovanov]

There is now a GameRunnerManager.parse_step_data() method that converts a database ModelRecord record into a dict representation.

It uses the corresponding *. get_data()` methods to retrieve the field data: https://github.com/kstaats/simoc/blob/abm_database/simoc_server/database/db_model.py#L163, https://github.com/kstaats/simoc/blob/abm_database/simoc_server/database/db_model.py#L191 https://github.com/kstaats/simoc/blob/abm_database/simoc_server/database/db_model.py#L232

[breathnach]

@Pebody Thanks for the changes and the code snippets. I have two questions below.

I notice GameRunnerManager.get_step now takes game_id as input, but views.get_step hasn't been updated to provide that. Should the game_id come from the front end or elsewhere?

Should I be calling GameRunnerManager.get_step_to() from views.get_step, before I try to access the data from any step?

[iurii-milovanov]

@breathnach you're right, you need to provide game_id as an argument to query any step data. That's the way how different ABM runs are distinguished in the db for a given user.

For a running simulation, you can get its game_id from the corresponding GameRunner object:

game_runner = game_runner_manager.get_game_runner(user)
game_id = game_runner.game_id

But I'm not sure that game_id should be tracked and handled by the backend. Although the backend still needs to send it back to the frontend once a new simulation is created.

What do you think?

Continued in: https://github.com/kstaats/simoc/issues/67#issue-446584131

[breathnach]

I've changed views.get_step so that it checks if game_id is given in the input, and if not is gets it from game_runner.game_id. If it's possible that there could be more than one game_id for a user, I think it would be worth requiring game_id from the front end.

On Mon, 20 May 2019 at 00:01, Iurii Milovanov notifications@github.com wrote:

@breathnach https://github.com/breathnach you're right, you need to provide game_id as an argument to query any step data. That's the way how different ABM runs are distinguished in the db for a given user.

For a running simulation, you can get its game_id from the corresponding GameRunner object:

game_runner = game_runner_manager.get_game_runner(user) game_id = game_runner.game_id

But I'm not sure that game_id should be tracked and handled by the backend. Although the backend still needs to send it back to the frontend once a new simulation is created.

What do you think?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kstaats/simoc/issues/65?email_source=notifications&email_token=ABMMAQUMW2FSGDLJAZTEQCDPWIWDHA5CNFSM4HNVCLUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVXWI4Q#issuecomment-493839474, or mute the thread https://github.com/notifications/unsubscribe-auth/ABMMAQVKBXPILASESXVOOELPWIWDHANCNFSM4HNVCLUA .

[breathnach]

@Pebody Is StorageCapacityRecord holding the capacities of the storages, or how much is actually in each storage per step?

i.e. should I be calculating the storage ratios per step from this or from the currency values in StepRecord.

[breathnach]

After implementing it does look like the info is in StorageCapacityRecord.

The comments of the get_step route include examples on how to determine which agents and currencies you want the storage ratios for.

The get_step function takes JSON input. The following is an example of what to include in the input to request storage_ratios, "storage_ratios":{"air_storage_1":["atmo_co2"]}. This results in the following added to the output JSON: "storage_ratios": {"air_storage_1": {"atmo_co2": 0.00038879091443387717}}

[iurii-milovanov]

Sorry for the late reply.

It seems that you've already figured this out, but just to confirm that StorageCapacityRecord table stores both "how much is actually in each storage per step" (value) and "the capacities of the storages" (capacity).

While the StepRecord table stores only what was sent to the storages, but it doesn't contain any information about the actual storage capacities.

[iurii-milovanov]

Hi Sinead,

Sorry for the delay in replying on this.

Just left you a comment in #65.

Thanks, Iurii

On Thu, 23 May 2019 at 16:23, breathnach notifications@github.com wrote:

After implementing it does look like the info is in StorageCapacityRecord.

The comments of the get_step route include examples on how to determine which agents and currencies you want the storage ratios for.

The get_step function takes JSON input. The following is an example of what to include in the input to request storage_ratios, "storage_ratios":{"air_storage_1":["atmo_co2"]}. This results in the following added to the output JSON: "storage_ratios": {"air_storage_1": {"atmo_co2": 0.00038879091443387717}}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kstaats/simoc/issues/65?email_source=notifications&email_token=ABOD4BKNHUKQX356H4OMCTLPW4DNPA5CNFSM4HNVCLUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWDQ5JA#issuecomment-495390372, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOD4BO3MQCC5CYRIQHQOP3PW4DNPANCNFSM4HNVCLUA .