Caribou-Go: Moving Monte-Carlo Simulation to Go

[pjavanrood]

Moving Monte-Carlo Simulation to Go, closes:

• [x] closes #283

This PR has two main components:

• caribou-go/
• GoDeploymentMetricsCalculator

caribou-go/ mirrors the functionality of the SimpleDeploymentMetricsCalculator.

The Go source code is compiled into a shared library, allowing it to be called from Python. The integration between Go and Python is managed through the GoDeploymentMetricsCalculator.

Data is passed between the Python and Go components using named pipes. On the Python side, this communication is handled by GoDeploymentMetricsCalculator, while on the Go side, it's managed in main/deploymentcalculator.go.

[github-actions[bot]]

Coverage Report

File	Stmts	Miss	Cover	Missing
caribou/common
utils.py	37	8	78%	43–58
caribou/common/models
endpoints.py	34	3	91%	18–19, 50
caribou/common/models/remote_client
aws_remote_client.py	478	66	86%	42, 65, 105, 151, 216, 252, 269–270, 298–299, 338, 349, 359, 392–394, 402–403, 422–424, 443, 467, 480–481, 484, 490, 530–532, 570–572, 590–591, 632, 638, 648, 652, 663–664, 672–673, 682, 686, 697, 707–709, 729, 740–742, 762, 777–779, 794–799, 827–828, 832, 848–852, 858–859
remote_client_factory.py	21	3	86%	21–23
caribou/data_collector/components
data_collector.py	18	1	94%	23
data_exporter.py	34	2	94%	49, 56
caribou/data_collector/components/carbon
carbon_collector.py	21	2	90%	34–35
carbon_retriever.py	134	18	87%	25, 59, 93, 135–144, 148, 153, 160, 168–169, 215, 246, 249, 256
caribou/data_collector/components/performance
performance_retriever.py	38	6	84%	44, 50–51, 56–59
caribou/data_collector/components/provider
provider_collector.py	24	2	92%	43–44
provider_retriever.py	294	50	83%	24, 32, 34, 47, 59, 77, 84, 100, 132–152, 183–281, 299–304, 336, 344, 346, 348, 350, 352, 354, 356, 432, 446, 556, 649
caribou/data_collector/components/workflow
workflow_collector.py	25	1	96%	34
workflow_retriever.py	305	109	64%	216–257, 280, 330–337, 339, 371–414, 436, 467–526, 533–563, 566–583, 587–604, 615, 676, 698
caribou/data_collector/utils/latency_retriever
aws_latency_retriever.py	74	9	88%	59, 63, 65, 67, 70, 74, 76, 78, 81
latency_retriever.py	6	1	83%	8
caribou/deployment/client
caribou.py	16	1	94%	22
caribou_workflow.py	465	135	71%	54, 70–71, 80–83, 129, 201–223, 267–268, 282, 315–326, 347–354, 380–391, 434–454, 562, 578, 581, 636–662, 667–683, 705–715, 793, 796, 798, 800, 802, 804, 831, 866–881, 916–917, 927–931, 961, 996, 1004, 1016–1096, 1109–1138, 1172–1181, 1212, 1245, 1273–1282, 1302–1320, 1340, 1353–1359, 1365
caribou/deployment/common/config
config.py	83	1	99%	104
caribou/deployment/common/deploy
deployer.py	138	16	88%	43, 94–101, 187, 228–230, 237, 244, 255
deployment_packager.py	192	45	77%	31, 34, 39, 63–70, 73–84, 90, 96, 101, 106, 128, 156, 169, 182, 190, 251–253, 259–274, 284, 286, 288, 291
executor.py	53	4	92%	32–33, 66, 68
workflow_builder.py	171	13	92%	43, 59, 146, 178, 186, 189, 191, 196, 205, 208, 210, 324, 330
caribou/deployment/common/deploy/models
function.py	41	2	95%	40, 51
function_instance.py	12	2	83%	16, 22
iam_role.py	34	3	91%	19–21
instructions.py	18	2	89%	17, 26
resource.py	18	1	94%	25
workflow.py	192	32	83%	37, 42, 64, 74, 96, 117, 163, 168, 203, 216–228, 269, 278, 286, 290, 295–324
caribou/deployment/common/deploy_instructions
aws_deploy_instructions.py	23	2	91%	35, 81
deploy_instructions.py	52	9	83%	57, 93, 143, 149, 162, 166, 170, 174, 180
caribou/deployment/common/factories
deploy_instruction_factory.py	18	3	83%	20–22
deployer_factory.py	107	16	85%	24–27, 41, 52–53, 82, 90, 99, 108, 119, 126–129
caribou/deployment/server
re_deployment_server.py	61	5	92%	36, 48, 51, 58, 107
caribou/deployment_solver
workflow_config.py	75	11	85%	24, 26, 32–33, 54–56, 61, 79, 82, 107
caribou/deployment_solver/deployment_algorithms
coarse_grained_deployment_algorithm.py	25	2	92%	27, 32
deployment_algorithm.py	98	11	89%	52, 91, 105, 118, 132, 149, 184–190, 219
fine_grained_deployment_algorithm.py	25	2	92%	38, 43
stochastic_heuristic_deployment_algorithm.py	92	14	85%	18–25, 49, 59, 71, 92, 95, 100, 106–109, 119, 136, 163, 170
caribou/deployment_solver/deployment_input
input_manager.py	199	55	72%	126, 132, 135, 194, 200, 246–254, 276–282, 291–294, 304–307, 320–339, 352–373, 379–387, 441–442, 476–487, 493–495, 501
caribou/deployment_solver/deployment_input/components/calculators
carbon_calculator.py	96	4	96%	124–128, 214, 261
cost_calculator.py	70	1	99%	163
runtime_calculator.py	109	11	90%	130–145, 152, 201, 209, 262, 295
caribou/deployment_solver/deployment_input/components/loaders
carbon_loader.py	21	3	86%	22, 43, 46
datacenter_loader.py	49	5	90%	106–111, 114
performance_loader.py	18	1	94%	38
region_viability_loader.py	17	2	88%	15, 32
workflow_loader.py	147	26	82%	86, 102–112, 122, 145, 165, 183, 193, 210, 264–276, 292, 364–369, 392, 430–432, 437
caribou/deployment_solver/deployment_metrics_calculator
deployment_metrics_calculator.py	40	1	98%	49
go_deployment_metrics_calculator.py	51	31	39%	18–21, 25–27, 41–50, 53, 64–73, 76–80, 83–85, 88–89
simple_deployment_metrics_calculator.py	129	17	87%	125–142, 170–171, 203–204, 220–223, 235–236, 251
caribou/deployment_solver/deployment_metrics_calculator/models
workflow_instance.py	194	82	58%	84–95, 107, 141–144, 183–186, 194–196, 203–227, 238–269, 275–289, 308–441, 520
caribou/deployment_solver/models
indexer.py	17	1	94%	25
caribou/deployment_solver/ranker
ranker.py	62	4	94%	14, 26–27, 95
caribou/endpoint
client.py	131	20	85%	46, 56, 104, 126–129, 140, 167, 183, 185, 222–226, 231–232, 236–239
caribou/monitors
deployment_manager.py	134	18	87%	64, 70, 81, 100, 122–127, 164–169, 184, 207, 222, 228, 263, 283
monitor.py	8	1	88%	12
caribou/syncers
log_sync_workflow.py	521	290	44%	108, 163–173, 178, 186, 193, 203, 210, 248–254, 256, 258, 260, 262, 264, 268, 272, 274, 276, 279–285, 288, 292, 302–369, 378–385, 395–463, 472–475, 480–495, 506–597, 610–664, 674–700, 705–722, 727–744, 758, 765–788, 797–798, 808, 813–819, 846–847, 849–850, 855–856, 890–897, 901, 938, 940, 945, 979, 983–986, 994–1004, 1007–1013, 1016–1027, 1030
log_syncer.py	42	1	98%	46
caribou/syncers/components
execution_data.py	74	4	95%	76, 86, 129, 137
workflow_run_sample.py	83	6	93%	136, 145–149
TOTAL	6367	1197	81%

Tests	Skipped	Failures	Errors	Time
640	0 :zzz:	0 :x:	0 :fire:	19.903s :stopwatch:

[engshahrad]

I agree with Viktor. Let's merge the code to have just one repo. Having a separate repo and using it as a submodule adds unnecessary complexity. I was always under the impression that using a separate repo was for easy testing, but that we'll be integrating all code inside the main caribou repo.

[pjavanrood]

Thank you both @vGsteiger and @engshahrad for your comments, as you both mentioned, I have added the Go source code as well, and we will keep everything in the same repo. I'm sorry for the confusion.

[pjavanrood]

I am not very familiar with the Go language, so my review primarily focuses on the overall structure and its similarity to the Python implementation. I was able to successfully run the Go solver on my end, and I can confirm that the results are very similar to those of the Python solver. However, I did spot some minor issues, including one potential mistranslation in the carbon calculator.

Additionally, I was wondering if it's possible to modify the constant variables to load directly from constants.py. This would help avoid the need to update both the constants file and the individual Go file separately, improving maintainability. If you believe this is feasible and worthwhile, could you create an issue for this? There's no need to include this change in the current PR.

I also noticed that almost every Go function has a corresponding unit test, except for inputmanager.go. This might have been overlooked.

Overall, great work!

Thanks for your review, Daniel! I addressed the changes you mentioned and also created an issue for automatically syncing constants between Python and Go (#291 )