drniiken commented 1 year ago

Description

To ensure transparency and accuracy in billing, it is essential to collect and keep track of all resource usage from users, including compute time for simulation hours and S3 storage per user as well as egress costs for download of files. This information must be synchronized with what the customer has paid for, and the data must be stored permanently to accommodate the pay-per-use model.

To make this data accessible to all relevant parties, it will be integrated into multiple systems, including the webpage, product, and API, as well as potentially the finance department for billing.

## Tasks
- [x] ~~Potentail Enhancement: Monitor storage costs (e.g. via prometheus S3 exporter ?)~~
- [x] ~~Overload testing (DB,..)~~
- [x] ~~Review Pricing Data Backups @mrnicegyu11 @yuryhrytsuk~~
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1018
- [ ] ITISFoundation/osparc-issues#956
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4674
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1308
- [ ] https://github.com/ITISFoundation/osparc-ops-environments/issues/174
- [x] ~~Setup infrastructure for the second database~~
- [x] ~~Show credits burnt by the running project @odeimaiz @jsaq007~~
- [x] ~~Credits indicator: Bar, Text or Text with Bar to Preferences @odeimaiz~~

## Schoggilebe
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1133
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1170
- [x] Tagging for cost separation @matusdrobuliak66 @mrnicegyu11

### ThisIsSparta
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1214
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1242

### Kobayashi Maru
- [x] Fetching data when I click on the tabs in the billing center
- [x] https://github.com/ITISFoundation/osparc-issues/issues/1213
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1239

### Microhistory
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4475
- [x] ~~Create Admin/Accountant endpoint for downloading data & modifying prices~~
- [x] Have a script that collects all container seconds per user group for jupyter-smash in AWS prod @mrnicegyu11
- [x] ~~Connect External Service for User Credits~~
- [x] Credits indicator: yellow when it goes below 100$, red bellow 20$ @odeimaiz
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1004
- [x] ~~Inject AWS EC2 instance type to the container labels @sanderegg @matusdrobuliak66~~
- [x] Adding user-selected resoures (cpu/memory) labels to Computational services on start @sanderegg
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/4244
- [x] Adding user-selected resoures (cpu/memory) labels to Dynamic services on start  @sanderegg
- [x] ~~Inject project name/node label/user email to the container labels @sanderegg @matusdrobuliak66~~
- [x] Mechanism for handling lost heartbeats @matusdrobuliak66
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4958
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4768
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1131
- [x] get invoices (from the 3rd party payment system) @pcrespov
- [x] Few free welcome credits could be loaded in this private wallet (Show it in dollars, 100$ by default) @pcrespov
- [x] Tier selection, hide it if not Advanced @odeimaiz

### Quilmes
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1071
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1132
- [x] User's default wallet (on account creation) @pcrepov
- [x] Frontend: Show user credits @odeimaiz
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1142

### Baklava
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1051
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1007
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1067
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/697
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1070
- [x] Add tracking of iSolve @matusdrobuliak66
- [x] Start to trace additional services @matusdrobuliak66

### Sundae
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1039
- [x] Frontend: Show wallet info @odeimaiz
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/1019
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4523
- [x] Create User Credit History table @matusdrobuliak66
- [x] Create Credit table (mapping of instances, CPU hours to price) @matusdrobuliak66

### Watermelon
- [ ] ITISFoundation/osparc-issues#955
- [x] https://github.com/ITISFoundation/osparc-simcore/pull/4380 @matusdrobuliak66
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/994
- [ ] https://github.com/ITISFoundation/osparc-issues/issues/995
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4350
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/4453
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/3638
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/4322
- [ ] https://github.com/ITISFoundation/osparc-ops-environments/issues/175

sanderegg commented 1 year ago

Goal for sprint Jelly Beans

Use jsmash e2e to collect data (cpu/gpu hours, s3 storage, egress network)
- see what is missing.

mrnicegyu11 commented 1 year ago

Update for sprint Jelly Beans

We have a very rough PoC python script that:

gets service running time per user per service
gets service actual cpu usage per user per service
get service CPU reservation/limitation per user per service

Prometheus is used as a datasource. Potentially, this can serve as a basis for a new osparc-service that collects and exposes resource usage.

To make it robust, we need:

Wrap PoC in robust tested osparc code
Add a secondary Prometheus which is robustly set up (federated or separate, best managed prometheus from aws)
Add regular dumps of usage reports to file / S3 / backup for billing and storing these numbers at Z43. This needs to be reliable and tested
Potentially interface with postgres to get list of valid userids or resolve userID to email, etc.

The PoC script is found here https://github.com/ITISFoundation/osparc-simcore/pull/4168

mrnicegyu11 commented 1 year ago

Question from my side for the sprint planning PastelDeNata (I will be absent on PM2a):

Who is billed in case of unscheduled or scheduled maintenances? Imagine a solver job that is long running and killed due to an incident or scheduled downtime. How do we detect it inside our (to be devised) cost-system?
Is it truly necessary that we track additional metrics apart from simulation container seconds? It is at least conceivable to me to have a business model we charge the users a "flatrate" prices based only on simulation hours, that includes some additional charge for egress and S3 costs on our side as well. It would make the whole pipeline much smoother. I dont see value in devising a complicated system to track for example egress costs per user at this point (Personal opinion) . If a user needs to download a file for their scientific project, they are gonna do it anyway. A straight forward billing model might be easier to comprehend for end users. I doubt scientist want to even think about whether they cause egress or not. ---> I propose a flatrate charge based only on simulation seconds.

mguidon commented 1 year ago

Failed jobs will not be billed (we will need to make sure that we disallow starting jobs prior to a downtime). For iSolve it might be possible to estimate the runtime
We will have two scenarios:
1. Cloud users will be billed by simulation hours
2. desktop users that submit simluations will be billed by 100 + X % of our costs (ec2 + egress, egress could be tracked by API) In general, even if we do not bill it, I would like to know how much S3 storage a user consumes. Also I would like to know how much egress cost the video streaming costs (no on a per user basis). The latter is to get an idea of the cost and whether we should go for a AWS solution for webrtc.

pcrespov commented 1 year ago

Goal for sprint Pastel de Nata

Have a script that collects all container seconds per user group for jupyter-smash in AWS prod @mrnicegyu11
Do the same with iSolve @mrnicegyu11

sanderegg commented 1 year ago

Update for sprint Pastel de Nata

new service resource-usage-tracker ready to be used:
- https://github.com/ITISFoundation/osparc-simcore/pull/4244
- https://github.com/ITISFoundation/osparc-simcore/pull/4275

matusdrobuliak66 commented 1 year ago

Update Watermelon

Working comments

[x] adding needed labels to the computational services
[x] based on @GitHK input: Always check the last updated row -> and start the data fetching/populating of the database from there.
- https://github.com/ITISFoundation/osparc-simcore/pull/4420
[x] adding endpoint for frontend (so we can get quick feedback)
- https://github.com/ITISFoundation/osparc-simcore/pull/4445
[ ] adding useful CLI commands (for example during an outage to be able to run/rerun the concrete task at a specific time)
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/4380#discussion_r1237479486
[ ] adding an additional test that will test concurrency (inspired by test_multiple_creation_deletion_of_nodes)
[ ] IMPORTANT: adding a second data source from the application to get info when computational/dynamic services started/stopped.

Notes & Discussions:

as promql doesn't support pagination when querying, we need to think of a list of regex expressions that would cover containers we would like to monitor (which will fetch a reasonable amount of data) and we will run more background tasks for each item in the list
- OPEN DISCUSSION (with @mrnicegyu11): should we run all background tasks in one container, or should each background task run as a separate container in our docker swarm.
NOTE: potentially introduce a daily aggregated table which will aggregate the data needed for billing service (therefore we would be able to keep the _resource_trackercontainer table data amount under control.
NOTE: point of failure
- resource tracking service is unavailable -> The scheduled task in this service can run even backward, the only important thing is that the Prometheus and its database is available
- Prometheus outage -> There is a time range where we do not have data because Prometheus was not scraping them -> We will run at least 2 replicas of Prometheus in different nodes
- There should be another source of data! -> We will store when services were started and stopped from the application.

matusdrobuliak66 commented 1 year ago

ISSUES found on the way:

sometimes we have hanging sidecars, therefore containers stay running master: ex. project_id = 4517c942-1114-11ee-954c-02420a00044e, node_id = a796a9e0-2652-5eb4-8955-9219d77be36c, user = puppeter


_ports_outputs_pull.316d0aa2-3570-48c6-a4c4-087076a44121 HTTP/1.1" 200
log_level=INFO | log_timestamp=2023-06-22 15:49:01,197 | log_source=uvicorn.access:send(438) | log_uid=None | log_msg=172.13.0.4:56712 - "GET /task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_restore_state.c6fba4b4-d146-404b-b5b0-c6eba5ba411d HTTP/1.1" 200
log_level=INFO | log_timestamp=2023-06-22 15:49:01,212 | log_source=simcore_sdk.node_data.data_manager:_pull_file(136) | log_uid=None | log_msg=pulling data from 4517c942-1114-11ee-954c-02420a00044e/a796a9e0-2652-5eb4-8955-9219d77be36c/workspace.zip to /tmp/tmpxsws8st3/workspace.zip...
log_level=INFO | log_timestamp=2023-06-22 15:49:01,337 | log_source=simcore_service_dynamic_sidecar.modules.nodeports:download_target_ports(306) | log_uid=None | log_msg=Downloaded 0.0B in 0.1322210943326354 seconds

downloading /master-simcore/4517c942-1114-11ee-954c-02420a00044e/a796a9e0-2652-5eb4-8955-9219d77be36c/workspace.zip --> workspace.zip : 0%| | 0.00/12.6k [00:00<?, ?byte/s] downloading /master-simcore/4517c942-1114-11ee-954c-02420a00044e/a796a9e0-2652-5eb4-8955-9219d77be36c/workspace.zip --> workspace.zip : 100%|██████████| 12.6k/12.6k [00:00<00:00, 2.88Mbyte/s] log_level=INFO | log_timestamp=2023-06-22 15:49:01,374 | log_source=simcore_sdk.node_data.data_manager:_pull_file(149) | log_uid=None | log_msg=completed pull of /tmp/tmpxsws8st3/workspace.zip.

decompressing /tmp/tmpxsws8st3/workspace.zip -> /dy-volumes/home/jovyan/work/workspace [3 files/12.635KiB] : 0%| | 0.00/12.6k [00:00<?, ?file/s] decompressing /tmp/tmpxsws8st3/workspace.zip:jl_notebook.ipynb -> /dy-volumes/home/jovyan/work/workspace/jl_notebook.ipynb : 0%| | 0.00/9.18k [00:00<?, ?byte/s] decompressing /tmp/tmpxsws8st3/workspace.zip:jl_notebook.ipynb -> /dy-volumes/home/jovyan/work/workspace/jl_notebook.ipynb : 100%|██████████| 9.18k/9.18k [00:00<00:00, 40.6Mbyte/s]

decompressing /tmp/tmpxsws8st3/workspace.zip:.hidden_do_not_remove -> /dy-volumes/home/jovyan/work/workspace/.hidden_do_not_remove : 0%| | 0.00/227 [00:00<?, ?byte/s] decompressing /tmp/tmpxsws8st3/workspace.zip:.hidden_do_not_remove -> /dy-volumes/home/jovyan/work/workspace/.hidden_do_not_remove : 100%|██████████| 227/227 [00:00<00:00, 1.66Mbyte/s]

decompressing /tmp/tmpxsws8st3/workspace.zip:README.ipynb -> /dy-volumes/home/jovyan/work/workspace/README.ipynb : 0%| | 0.00/2.89k [00:00<?, ?byte/s] decompressing /tmp/tmpxsws8st3/workspace.zip:README.ipynb -> /dy-volumes/home/jovyan/work/workspace/README.ipynb : 100%|██████████| 2.89k/2.89k [00:00<00:00, 32.2Mbyte/s]

decompressing /tmp/tmpxsws8st3/workspace.zip -> /dy-volumes/home/jovyan/work/workspace [3 files/12.635KiB] : 100%|██████████| 12.6k/12.6k [00:00<00:00, 1.39Mfile/s] log_level=INFO | log_timestamp=2023-06-22 15:49:02,202 | log_source=uvicorn.access:send(438) | log_uid=None | log_msg=172.13.0.4:56722 - "GET /task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_outputs_pull.316d0aa2-3570-48c6-a4c4-087076a44121 HTTP/1.1" 200 log_level=INFO | log_timestamp=2023-06-22 15:49:02,203 | log_source=uvicorn.access:send(438) | log_uid=None | log_msg=172.13.0.4:56728 - "GET /task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_restore_state.c6fba4b4-d146-404b-b5b0-c6eba5ba411d HTTP/1.1" 200 log_level=INFO | log_timestamp=2023-06-22 15:49:02,210 | log_source=uvicorn.access:send(438) | log_uid=None | log_msg=172.13.0.4:56730 - "GET /task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_ports_outputs_pull.316d0aa2-3570-48c6-a4c4-087076a44121/result HTTP/1.1" 200 log_level=INFO | log_timestamp=2023-06-22 15:49:02,211 | log_source=uvicorn.access:send(438) | log_uid=None | log_msg=172.13.0.4:56742 - "GET /task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_restore_state.c6fba4b4-d146-404b-b5b0-c6eba5ba411d/result HTTP/1.1" 200 log_level=INFO | log_timestamp=2023-06-22 15:49:02,227 | log_source=uvicorn.access:send(438) | log_uid=None | log_msg=172.13.0.4:56750 - "POST /v1/containers/ports/outputs/dirs HTTP/1.1" 204 log_level=INFO | log_timestamp=2023-06-22 15:49:02,258 | log_source=uvicorn.access:send(438) | log_uid=None | log_msg=172.13.0.4:56762 - "POST /v1/containers HTTP/1.1" 202 log_level=INFO | log_timestamp=2023-06-22 15:49:02,263 | log_source=uvicorn.access:send(438) | log_uid=None | log_msg=172.13.0.4:56764 - "GET /task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_create_service_containers.43c42531-85c4-467d-a863-89bb4f5b1fae HTTP/1.1" 200 log_level=INFO | log_timestamp=2023-06-22 15:49:03,266 | log_source=uvicorn.access:send(438) | log_uid=None | log_msg=172.13.0.4:56778 - "GET /task/simcore_service_dynamic_sidecar.modules.long_running_tasks.task_create_service_containers.43c42531-85c4-467d-a863-89bb4f5b1fae HTTP/1.1" 200 log_level=INFO | log_timestamp=2023-06-22 15:49:03,638 | log_source=simcore_service_dynamic_sidecar.modules.long_running_tasks:task_create_service_containers(139) | log_uid=None | log_msg=Validated compose-spec:\nnetworks:\n back----end:\n internal: false\n dy-sidecar_a796a9e0-2652-5eb4-8955-9219d77be36c:\n driver: overlay\n external:\n name: dy-sidecar_a796a9e0-2652-5eb4-8955-9219d77be36c\n master-simcore_interactive_services_subnet:\n driver: overlay\n external:\n name: master-simcore_interactive_services_subnet\nservices:\n dy-sidecar-a796a9e0-2652-5eb4-8955-9219d77be36c-0-jupyter-smash:\n container_name: dy-sidecar-a796a9e0-2652-5eb4-8955-9219d77be36c-0-jupyter-smash\n cpus: 4.0\n environment:\n - DISPLAY=:0\n - DY_SIDECAR_PATH_INPUTS=/home/jovyan/work/inputs\n - DY_SIDECAR_PATH_OUTPUTS=/home/jovyan/work/outputs\n - DY_SIDECAR_STATE_PATHS=["/home/jovyan/work/workspace"]\n - SIMCORE_NANO_CPUS_LIMIT=4000000000\n - SIMCORE_MEMORY_BYTES_LIMIT=17179869184\n - SIMCORE_NODE_BASEPATH=\n - SYM_SERVERHOSTNAME=sym-server%service_uuid%\n - DY_BOOT_OPTION_BOOT_MODE=0\n - PUID=8004\n - PGID=8004\n image: registry.osparc-master.speag.com/simcore/services/dynamic/jupyter-smash:3.0.7\n init: true\n labels:\n - product_name=osparc\n - simcore_user_agent=puppeteer\n - study_id=4517c942-1114-11ee-954c-02420a00044e\n - user_id=727\n - uuid=a796a9e0-2652-5eb4-8955-9219d77be36c\n mem_limit: '17179869184'\n mem_reservation: '536870912'\n networks:\n back----end: null\n dy-sidecar_a796a9e0-2652-5eb4-8955-9219d77be36c: null\n volumes:\n - /tmp/.X11-unix:/tmp/.X11-unix\n - /docker/volumes/dyv_25238755-5be8-4d92-b501-9c0750630d59_a796a9e0-2652-5eb4-8955-9219d77be36c_stupni_krow_nayvojemoh/_data:/home/jovyan/work/inputs\n - /docker/volumes/dyv_25238755-5be8-4d92-b501-9c0750630d59_a796a9e0-2652-5eb4-8955-9219d77be36c_stuptuo_krow_nayvojemoh/_data:/home/jovyan/work/outputs\n - /docker/volumes/dyv_25238755-5be8-4d92-b501-9c0750630d59_a796a9e0-2652-5eb4-8955-9219d77be36c_ecapskrow_krow_nayvojemoh/_data:/home/jovyan/work/workspace\nversion: '2.3'\n log_level=INFO | log_timestamp=2023-06-22 15:49:09,223 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/workspace/.hidden_do_not_remove': permissions=-rw-r--r-- uid=1000 gid=8004 size=227.0B\nFile stat: os.stat_result(st_mode=33188, st_ino=306149258, st_dev=2049, st_nlink=1, st_uid=1000, st_gid=8004, st_size=227, st_atime=1687448940, st_mtime=1687448941, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,224 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/workspace/jl_notebook.ipynb': permissions=-rw-r--r-- uid=1000 gid=8004 size=9.2KiB\nFile stat: os.stat_result(st_mode=33188, st_ino=306149259, st_dev=2049, st_nlink=1, st_uid=1000, st_gid=8004, st_size=9404, st_atime=1687448941, st_mtime=1687448941, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,224 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/workspace/README.ipynb': permissions=-rw-r--r-- uid=1000 gid=8004 size=2.9KiB\nFile stat: os.stat_result(st_mode=33188, st_ino=306149273, st_dev=2049, st_nlink=1, st_uid=1000, st_gid=8004, st_size=2957, st_atime=1687448941, st_mtime=1687448941, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,224 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/workspace': permissions=drwxrwxr-- uid=1000 gid=998 size=80.0B\nFile stat: os.stat_result(st_mode=16892, st_ino=306220969, st_dev=2049, st_nlink=2, st_uid=1000, st_gid=998, st_size=80, st_atime=1687448949, st_mtime=1687448941, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,225 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/inputs/.hidden_do_not_remove': permissions=-rw-r--r-- uid=1000 gid=8004 size=228.0B\nFile stat: os.stat_result(st_mode=33188, st_ino=306149257, st_dev=2049, st_nlink=1, st_uid=1000, st_gid=8004, st_size=228, st_atime=1687448940, st_mtime=1687448940, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,225 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/inputs': permissions=drwxrwxr-- uid=1000 gid=998 size=35.0B\nFile stat: os.stat_result(st_mode=16892, st_ino=306220968, st_dev=2049, st_nlink=2, st_uid=1000, st_gid=998, st_size=35, st_atime=1687448949, st_mtime=1687448940, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,226 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/outputs/.hidden_do_not_remove': permissions=-rw-r--r-- uid=1000 gid=8004 size=228.0B\nFile stat: os.stat_result(st_mode=33188, st_ino=2550292891, st_dev=2049, st_nlink=1, st_uid=1000, st_gid=8004, st_size=228, st_atime=1687448940, st_mtime=1687448940, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,226 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/outputs/key_values.json': permissions=-rw-r--r-- uid=1000 gid=8004 size=192.0B\nFile stat: os.stat_result(st_mode=33188, st_ino=2550292892, st_dev=2049, st_nlink=1, st_uid=1000, st_gid=8004, st_size=192, st_atime=1687448941, st_mtime=1687448941, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,226 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/outputs/output_1': permissions=drwxr-xr-x uid=1000 gid=8004 size=6.0B\nFile stat: os.stat_result(st_mode=16877, st_ino=3565018646, st_dev=2049, st_nlink=2, st_uid=1000, st_gid=8004, st_size=6, st_atime=1687448942, st_mtime=1687448942, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,226 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/outputs/output_2': permissions=drwxr-xr-x uid=1000 gid=8004 size=6.0B\nFile stat: os.stat_result(st_mode=16877, st_ino=283256553, st_dev=2049, st_nlink=2, st_uid=1000, st_gid=8004, st_size=6, st_atime=1687448942, st_mtime=1687448942, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,226 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/outputs/output_3': permissions=drwxr-xr-x uid=1000 gid=8004 size=6.0B\nFile stat: os.stat_result(st_mode=16877, st_ino=1429624200, st_dev=2049, st_nlink=2, st_uid=1000, st_gid=8004, st_size=6, st_atime=1687448942, st_mtime=1687448942, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,226 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/outputs/output_4': permissions=drwxr-xr-x uid=1000 gid=8004 size=6.0B\nFile stat: os.stat_result(st_mode=16877, st_ino=2550292893, st_dev=2049, st_nlink=2, st_uid=1000, st_gid=8004, st_size=6, st_atime=1687448942, st_mtime=1687448942, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,226 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/outputs': permissions=drwxrwxr-- uid=1000 gid=998 size=122.0B\nFile stat: os.stat_result(st_mode=16892, st_ino=2550448079, st_dev=2049, st_nlink=6, st_uid=1000, st_gid=998, st_size=122, st_atime=1687448949, st_mtime=1687448942, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,228 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/workspace/README.ipynb': permissions=-rw-rw-rw- uid=1000 gid=100 size=2.9KiB\nFile stat: os.stat_result(st_mode=33206, st_ino=306149253, st_dev=2049, st_nlink=1, st_uid=1000, st_gid=100, st_size=2957, st_atime=1661260111, st_mtime=1661260111, st_ctime=1687448949) log_level=INFO | log_timestamp=2023-06-22 15:49:09,228 | log_source=simcore_service_dynamic_sidecar.modules.attribute_monitor._logging_event_handler:event_handler(34) | log_uid=None | log_msg=Attribute change to: '/dy-volumes/home/jovyan/work/workspace/README.ipynb': permissions=-rw-rw-rw- uid=1000 gid=100 size=2.9KiB\nFile stat: os.stat_result(st_mode=33206, st_ino=306149253, st_dev=2049, st_nlink=1, st_uid=1000, st_gid=100, st_size=2957, st_atime=1661260111, st_mtime=1661260111, st_ctime=1687448949) log_level=WARNING | log_timestamp=2023-06-22 15:55:00,326 | log_source=servicelib.long_running_tasks._task:_stale_tasks_monitor_worker(119) | log_uid=None | log_msg=Removing stale task 'simcore_service_dynamic_sidecar.modules.long_running_tasks.task_create_service_containers.43c42531-85c4-467d-a863-89bb4f5b1fae' with status '{"task_progress": {"message": "finished", "percent": 1.0}, "done": true, "started": "2023-06-22T15:49:02.249855"}'

mguidon commented 1 year ago

Next iteration:

Track instance type
Have a table in the backend that maps instance type to credits/hour per product
Provide API for
- Available Credits left
- Update of Credit Count after purchase

Do this for comp. service iSolve and sim4life and jupyter-smash

matusdrobuliak66 commented 1 year ago

IMPLEMENTATION NOTES:

PDF: _5e3d5799c2f3370c861d6889-Implementation Notes - Resource Usage Tracker-200723-065246.pdf

original (needed access): https://osparc.atlassian.net/wiki/spaces/~5e3d5799c2f3370c861d6889/pages/327681/Implementation+Notes+-+Resource+Usage+Tracker

matusdrobuliak66 commented 1 year ago

Goal for sprint Sundae

Next iteration:

Introduce Wallets
Have a table in the backend that maps sim4life tier to credits/hour per product
Provide API for
- Available Credits left
- Update of Credit Count after purchase
Track sim4life from director-v2 tier chosen by the user [small, medium, high]

Do this for comp. service iSolve and sim4life and jupyter-smash

matusdrobuliak66 commented 1 year ago

Update Sundae

Done:

introduced wallets to osparc
discussed architecture of Usage tracking & Billing
publishing events to RabbitMQ from computation services
listening to the events in the resource usage tracker
created DB model for usage tracking & credits
created AWS managed RabbitMQ service (not yet used by simcore)

In progress:

storing events to the usage tracking table
publishing events to RabbitMQ from dynamic services

To do:

connect simcore to AWS managed RabbitMQ service
create all the logic around credits
- ex. monitor and update credits used
- propagate credit transactions to the frontend ...

Design:

Pricing plan DB design: pricing-plan-db-design.pdf

matusdrobuliak66 commented 1 year ago

Brainstorming with @pcrespov

What happens when we hit zero credits?

sim4life (dynamic service)
- Close the service ("maintenance style")
- running isolve (computational service) from the service -> what then?
run isolve with the API
- dumps current state and continues after user adds more credits (Docker freeze container functionality - docker pause & docker unpause)
- kill job right away
- Leave it running with some threshold (for example 10% of used credits in that job)
Accountant removes user from the wallet (unauthorized)
- kill job right away

Batch approach:

To ensure that at least some jobs finish, we should have some reasonable max number of jobs that we allow to run in parallel

matusdrobuliak66 commented 1 year ago

Update Baklava

Done:

User center dashboard
Collecting and storing events/resources now from all running services (Computational & Dynamic)
Created core backend for credit computation/mapping credits to pricing plans
Expose top-up credit endpoint to Payment Service

In progress:

Expose available credit information to the frontend
Provide wallet_id (where to bill) and pricing_plan_id (what to bill) when starting a service via UI or API.

To do:

Add a mechanism to close the running status of the service, if the resource tracker will stop recieving heartbeat events
Stop the running service if there are not enough credits
Connect simcore to AWS-managed RabbitMQ service
Connect resource usage tracker to new AWS-managed Postgres database ...

matusdrobuliak66 commented 11 months ago

Update Quilmes / TheNameless

Done:

Showing real credits in the wallet endpoint
Sending updates on current available credits in wallets to frontend via websocket
Wallet constrained to the specific product (example: osparc, s4l, ...)
User default wallet
Defining and listing pricing plans

In progress/To do:

API for storing the selected pricing plan for user service
Extend computation backend to use selected pricing plans
Introduce the lost heartbeat mechanism in RUT (to close the transaction if we lose the heartbeat)
e2e
Send to the RabbitMQ exchange that there are no credit left.

matusdrobuliak66 commented 10 months ago

Update Microhistory

Done and working

Selection of default wallet and pricing plan in the backend if not provided
Emit message when 0 credits are reached
Do not allow to start services when 0 credits are reached
Default wallet endpoint in the webserver
one-time-payment

should work in 3 weeks

Mechanism for handling lost heartbeat (almost done)
Billing data in the second database (if there is time left)
Admin endpoint for creation of pricing plans/units (if there is time left)

should not be available in 3 weeks

Optimization to have "Incremental SUM of credits table" so we do not need to always compute the whole history, which long term may get slow

matusdrobuliak66 commented 7 months ago

Update ThisIsSparta

Done

All known issues with computational backend sending events were fixed and now we are collecting all running jobs correctly.
https://github.com/ITISFoundation/osparc-issues/issues/1214
https://github.com/ITISFoundation/osparc-issues/issues/1242
https://github.com/ITISFoundation/osparc-simcore/pull/5285 (Burning credits now also works for different products, ex. academia, desktop,...)

Improvements to be done (not blocking release)

https://github.com/ITISFoundation/osparc-issues/issues/1133
"Incremental SUM" of credits (Currently when computing credits on the fly, we are counting the whole history of transactions, in the long term this is unsustainable and we need to compute credits incrementally)
Continually bill while the transaction is running (Currently we consider the transaction billed only when it is successfully finished)

ITISFoundation / osparc-issues

sim4life.io - WP3: Tracking of resource usage #922

Description

Goal for sprint Jelly Beans

Update for sprint Jelly Beans

Goal for sprint Pastel de Nata

Update for sprint Pastel de Nata

Update Watermelon

Working comments

ISSUES found on the way:

Goal for sprint Sundae

Update Sundae

Update Baklava

Update Quilmes / TheNameless

Update Microhistory

Done and working

should work in 3 weeks

should not be available in 3 weeks

Update ThisIsSparta

Done

Improvements to be done (not blocking release)