How to look at calculate throughput and delay of the system in CherrySim

[mabner1996]

Hello, I would like to ask how to use cherrysim to test the throughput of our device setup and the end-to-end delay.

I was thinking of looking at the debug module specifically the flood command (for throughput) and the ping command (for delay).

But perhaps there is a more efficient method to test this? Thanks you

[mariusheil]

I think I would need some more information on what exactly you want to test.

• The throughput and delay in CherrySim will be quite close to what you can get in reality but we have not "calibrated" all the simulations with real world measurements so you will be in the same ballpark at least.
• You can use commands such as the flood command for testing this, but better would be to use your actual implementation and, if sensors are attached, write mocks for these and generate the data in similar intervals as your sensors would do. This will give you the best estimate.
• Use the native renderer to have a nice visualization of the data flow
• Use the error log or the sim statistics (e.g. SIMSTATCOUNT) to collect some data about your setp.
• You should probably import your devices as a json with positions or you can also write a test in CherrySimTester to set them up using the predefined positions.
• You could script a testcase to collect what you need so you can always easily rerun it, which is better than doing it manually in CherrySimRunner

Marius

[mabner1996]

I am trying to simulate the test result I was doing before using fruitymesh. So its true that I am trying to use cherrsymtester, upload a Json file, and run a script to test it. Then, flood data from sensors every 3 minutes. I wanted to see whats the throughput and delay during the flooding of the data at this part.

In addition, I wanted to change the scanning and connection interval and see how it affects the throughputs and delay in the same settings.

But I do not get about the sim statistic part, how to implement it into the code?

Thank you, Michael

[mabner1996]

Another question is how do I use json file in the cherrysim tester

I tried to use the example in test_clustering.cpp but by uploading my own json file

TEST_P(MultiStackFixture, MyOwnCustomTest3) { std::string site = CherrySimUtils::GetNormalizedPath() + "/test/res/airport/site.json"; std::string device = CherrySimUtils::GetNormalizedPath() + "/test/res/airport/devices.json"; // We do not care for the exact clustering time as we only want to test that the jsons are still valid DoClusteringTestImportedFromJson(site, device, 1, 60 * 1000, 100000, GetParam()); }

but it creates some error, I attached the devices and site json file here too

airport.zip

[mabner1996]

the idea is that I want to count the throughput and delay from the closest to the furthest node

I will have a series of nodes in a line, I want to know those measurements from one end to another when I am flooding the mesh every 3 minutes.

Any help or suggestion?

[siretty]

Thank you for your interest in FruityMesh! I'm very sorry for the delayed response.

Regarding the exception: it is a result of your devices.json file specifying the featureset github_nrf52 instead of github_dev_nrf52 (or leaving it out altogether). See the example file in test/res/github_example/devices.json for one which uses github_dev_nrf52. If you don't specify a value for the cherrySimFeatureSet key in the devices.json the simulator will choose an appropriate featureset for you (which will be github_dev_nrf52).

Regarding the throughput test: you can take a look at TestThroughput in TestOther.cpp (at the bottom of the file). Here you find a usecase of the 'flood' functionality of the DebugModule.

Regarding the simulator statistics: you can take a look at the DISABLED_TestNoPacketsDropped in TestOther.cpp (currently disabled because of an unrelated bug which will be fixed in the next GitHub release). This requires that you insert a call to SIMSTATCOUNT("yourSimulatorStatisticKey"); into the place of the firmware you're interested in. It will increment a counter each time the line is hit during execution. The counter can be retrieved via a call to int errors = sim_get_statistics("yourSimulatorStatisticKey");. If you use this feature, be aware that the counter is global, i.e. if the SIMSTATCOUNT line is executed on multiple nodes the counter will accumulate the increments of all nodes the line is executed for. For some one-off measurements you could e.g. enclose the call to SIMSTATCOUNT in an if which checks if the current node has a particular node id (hint: cherrySimInstance->currentNode->GetNodeId()).

The 'simulator statistics' method is likely the most flexible way to accumulate custom counters.

Kind regards, Daniel

[mabner1996]

Hello, thank you for the reply.

I am currently trying to use the json file and trying to run the terminal command. Here is the code I am using

---

const int maxClusteringTimeMs = 1500 1000 1000; std::string site = CherrySimUtils::GetNormalizedPath() + "/test/res/airport/site.json"; std::string device = CherrySimUtils::GetNormalizedPath() + "/test/res/airport/devices.json"; CherrySimTesterConfig testerConfig = CherrySimTester::CreateDefaultTesterConfiguration(); SimConfiguration simConfig = CherrySimTester::CreateDefaultSimConfiguration();

simConfig.seed = 1;
simConfig.terminalId = 0;
simConfig.importFromJson = true;
simConfig.siteJsonPath = site;
simConfig.devicesJsonPath = device;
testerConfig.verbose = true;
//simConfig.nodeConfigName.insert( { "prod_mesh_nrf52", 10 } );
CherrySimTester tester = CherrySimTester(testerConfig, simConfig);

tester.Start();

---

as you can see I am using (simConfig.terminalId = 0;) so I can input terminal command afterwards, but I found out that the clustering cannot be done the terminal keep disconnected.

I can do the clustering if I change the simConfig.terminalId = -1; however I can't put any commands afterwards.

is there any way that I can keep the clustering while allowing to input the terminal commands? because the flood command will also need to be inputted with terminal

Thank you

[siretty]

I slightly adapted your test into the following (not using the MultiStackFixture which should not be relevant for your purposes):

TEST(TestClustering, MyOwnCustomTest3)
{
    std::string site   = CherrySimUtils::GetNormalizedPath() + "/test/res/airport/site.json";
    std::string device = CherrySimUtils::GetNormalizedPath() + "/test/res/airport/devices.json";

    CherrySimTesterConfig testerConfig = CherrySimTester::CreateDefaultTesterConfiguration();
    testerConfig.verbose = true;

    SimConfiguration simConfig = CherrySimTester::CreateDefaultSimConfiguration();
    simConfig.importFromJson = true;
    simConfig.siteJsonPath = site;
    simConfig.devicesJsonPath = device;
    simConfig.terminalId = 0;

    CherrySimTester tester = CherrySimTester(testerConfig, simConfig);

    tester.Start();
    tester.SimulateUntilClusteringDone(100000);
    printf("Clustering Time: %zu\n", tester.sim->simState.simTimeMs);
}

This has all logging enabled and waits for clustering to complete.

[siretty]

Hello, I'll close this as there has not been any more feedback. Feel free to reopen it.