Not able to run test

[TRMir]

Hello, I am writing a bachelor thesis and I want to use this software in it. My goal is to simulate and test some scenarios of InterUSS and use these results in my thesis. I intend to use only one local machine for this. I am running Win 10.

I am trying to run this through Docker. I download the repository using GitHub and go to build and dev folder where I use the run_locally shell script. The script runs and creates several images and a container with five sub-containers in it. However, at the end of the script I get the message that:

"scd_bootstrapper-1 exited with code 1 rid_bootstrapper-1 exited with code 1 dss-1 | exec /startup/core_service.sh: no such file or directory dss-1 exited with code 1"

I checked the startup files and all files are present. When trying to run the container, only the oauth-1 and crdb-1 are running, the rest of them exit with code 1. I am unsure of how to resolve this problem. I tried following these instructions on how to run a local test but have been unsuccessfull so far (https://github.com/interuss/monitoring/blob/main/monitoring/uss_qualifier/local_testing.md). Can someone help me with this?

Thank you in advance.

[BenjaminPelletier]

and go to build and dev folder where I use the run_locally shell script.

Which instructions are you attempting to follow? The local_testing.md document you mentioned and the quick start documentation for uss_qualifier use make commands to deploy the infrastructure.

dss-1 | exec /startup/core_service.sh: no such file or directory

It appears that this container is not mounting the /startup folder properly because that's where core_service.sh is.

I checked the startup files and all files are present.

Where did you look for startup files and how did you determine they were present? The thing we care about based on the error message above is that core_service.sh is placed in the /startup folder of the DSS container. Since the error message is that that file wasn't found, presumably it isn't getting into that folder in the container. The intent is that this is accomplished by the volume mount linked above. There shouldn't really be much room for things to go wrong -- what are you using to run scripts (cygwin? sh.exe? in a command prompt, Powershell, etc?), what command are you running, and what working folder are you running it from?

[TRMir]

Sorry for the late reply, I was trying out a few solutions. Just to clarify my previous issue:

I downloaded the entire monitoring repository using git. After downloading it, I tried to run the shell script in the folder /build/dev. I did not run it through a terminal, just by clicking it. When the script finished, it was not able to install 3 of the 5 subcontainers properly. It was giving me error code 1 and claiming that the files in the /startup folder were not found. I checked that the files were there by accesing the locally downloaded repository. The repository was downloaded to my /user/ folder on my pc.

I was trying to follow this : https://github.com/interuss/monitoring/blob/main/monitoring/README.md [nteroperability equipment]

My ultimate goal is to run DSS load test. I partially resolved the issue described above by installing Ubuntu Jammy and installing Docker CE. It no longer gives me error 1. However, now the local_infra-dss-1 exits with code one error. Code from make start-locally below:

tomas@tomas-All-Series:~$ cd monitoring tomas@tomas-All-Series:~/monitoring$ make start-locally build/dev/run_locally.sh up -d WARN[0000] /home/tomas/monitoring/build/dev/docker-compose.yaml:version` is obsolete [+] Running 23/16 ✔ scd_bootstrapper Pulled 55.2s ✔ dss Pulled 56.2s ✔ crdb Pulled 90.8s ✔ rid_bootstrapper Pulled 59.1s ✔ oauth Pulled 26.0s

[+] Running 8/8 ✔ Network interop_ecosystem_network Created 1.2s ✔ Network dss_internal_network Created 1.1s ✔ Volume "local_infra_dss_component_coordination" Created 1.5s ✔ Container local_infra-crdb-1 Started 41.2s ✔ Container local_infra-oauth-1 Started 40.1s ✔ Container local_infra-scd_bootstrapper-1 Started 54.7s ✔ Container local_infra-rid_bootstrapper-1 Started 56.0s ✔ Container local_infra-dss-1 Started 70.3s build/dev/wait_for_local_infra.sh Waiting for local interoperability ecosystem infrastructure (DSS & OAuth) to become available... Waiting for local_infra-dss-1 to be available (currently starting)................................Waiting for local_infra-dss-1 to be available (currently unhealthy)............Timeout reached. Container failed to become available. Exiting. make: *** [Makefile:65: start-locally] Error 1`

For the LoadTest tool i follow steps 1 and 3 in Running from container and that runs successfully. However, I cannot connect to the DSS instance port as described here: For the Host, provide the DSS Core Service endpoint used for testing. An example of such url is: http://local_infra-dss-1:8082/v1/dss/ in case local environment is setup by [run_locally.sh](https://github.com/interuss/monitoring/blob/main/build/dev/run_locally.sh) All of the requests have a 100% failure rate.

Could you please tell me what am I doing wrong?

Thank you.

[BenjaminPelletier]

I tried to run the shell script in the folder /build/dev. I did not run it through a terminal, just by clicking it.

I suspect Windows may have unexpected semantics associated with running a script by clicking. But best success is usually achieved by following established instructions (which use make and do not involve clicking) and it looks like you're switching to those, so I'll focus on the standard make-based approach.

My ultimate goal is to run DSS load test.

Sounds good -- in that case, you'll only need (as a prerequisite) to bring up a local interoperability ecosystem with an auth server (Dummy OAuth) and a DSS instance, and this is accomplished in one command the way you're attempting: make start-locally

The logs you provided are good -- the warning about version being obsolete is perhaps something to look at more closely later, but I don't think it's the cause of the immediate problem. The immediate problem is that wait_for_local_infra.sh has determined that the DSS core service container (local_infra-dss-1) didn't start successfully. Presumably (but useful to verify), docker container ls will not list local_infra-dss-1 as a running container, but docker container ls -a will show that that container exited. We want to know why it exited (or why it isn't responding, if it's still running), so please provide the logs of that container with docker container logs local_infra-dss-1. It may be useful to see the logs for the two database bootstrapper containers as well (local_infra-scd_bootstrapper-1 and local_infra-rid_bootstrapper-1).

We can't actually troubleshoot any further without a working DSS instance (and of course any attempt at a load test for a service that isn't running is going to produce 100% failure), but to get ahead of what I suspect we will need to troubleshoot next, the Use instructions you're referring to require that a load test server be running. Which instructions did you use to run the load test server? Run locally without Docker or Running in a Container? For either, which option did you choose?

[TRMir]

On a second try, it seems to set up itself correctly. Here are the logs: ` tomas@tomas-All-Series:~$ cd monitoring

tomas@tomas-All-Series:~/monitoring$ make start-locally build/dev/run_locally.sh up -d WARN[0000] /home/tomas/monitoring/build/dev/docker-compose.yaml: version is obsolete [+] Running 5/5 ✔ Container local_infra-oauth-1 Started 1.6s ✔ Container local_infra-crdb-1 Started 1.4s ✔ Container local_infra-rid_bootstrapper-1 Started 4.1s ✔ Container local_infra-scd_bootstrapper-1 Started 4.0s ✔ Container local_infra-dss-1 Started 7.4s build/dev/wait_for_local_infra.sh Waiting for local interoperability ecosystem infrastructure (DSS & OAuth) to become available... Waiting for local_infra-dss-1 to be available (currently starting)............ Local interoperability ecosystem infrastructure is now available.

tomas@tomas-All-Series:~/monitoring$ docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES feee44d3d7f2 interuss/dss:v0.15.0-rc1 "/startup/core_servi…" 21 hours ago Up 2 minutes (healthy) 0.0.0.0:4000->4000/tcp, 8082/tcp, 0.0.0.0:8082->80/tcp local_infra-dss-1 f2556c77d5c7 cockroachdb/cockroach:v21.2.7 "/cockroach/cockroac…" 21 hours ago Up 2 minutes 0.0.0.0:8080->8080/tcp, 0.0.0.0:26257->26257/tcp local_infra-crdb-1 1a509092724a interuss/dummy-oauth "/usr/bin/dummy-oaut…" 21 hours ago Up 2 minutes 0.0.0.0:8085->8085/tcp local_infra-oauth-1

tomas@tomas-All-Series:~/monitoring$ docker container ls -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES feee44d3d7f2 interuss/dss:v0.15.0-rc1 "/startup/core_servi…" 21 hours ago Up 2 minutes (healthy) 0.0.0.0:4000->4000/tcp, 8082/tcp, 0.0.0.0:8082->80/tcp local_infra-dss-1 61866062506d interuss/dss:v0.15.0-rc1 "/startup/scd_bootst…" 21 hours ago Exited (0) 2 minutes ago local_infra-scd_bootstrapper-1 93547a04cced interuss/dss:v0.15.0-rc1 "/startup/rid_bootst…" 21 hours ago Exited (0) 2 minutes ago local_infra-rid_bootstrapper-1 f2556c77d5c7 cockroachdb/cockroach:v21.2.7 "/cockroach/cockroac…" 21 hours ago Up 2 minutes 0.0.0.0:8080->8080/tcp, 0.0.0.0:26257->26257/tcp local_infra-crdb-1 1a509092724a interuss/dummy-oauth "/usr/bin/dummy-oaut…" 21 hours ago Up 2 minutes 0.0.0.0:8085->8085/tcp local_infra-oauth-1

tomas@tomas-All-Series:~/monitoring$ docker container logs local_infra-dss-1 Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. SCD pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. Waiting on DB bootstrapping... RID pending. DB bootstrap complete; starting core service... Debug Mode: off 2024-05-22T16:11:52.235Z info core-service/main.go:210 build {"address": ":80", "description": {"Time":"2024-05-03.13:05:22","Commit":"72367c9-dirty","Host":"buildkitsandbox"}} 2024-05-22T16:11:52.235Z info core-service/main.go:211 config {"address": ":80", "scd": true} 2024-05-22T16:11:52.251Z info core-service/main.go:319 Starting DSS HTTP server {"address": ":80"} 2024-05-22T16:12:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:12:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:13:52.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:13:52.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:14:52.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:14:52.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:15:52.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:15:52.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:16:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:16:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:17:52.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:17:52.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:18:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:18:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:19:52.000Z warn core-service/main.go:78 Failed periodic DB Ping (TotalConns=0) {"Database": "scd"} 2024-05-22T16:19:52.000Z warn core-service/main.go:78 Failed periodic DB Ping (TotalConns=0) {"Database": "rid"} 2024-05-22T16:20:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:20:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:21:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:21:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:22:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:22:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:23:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:23:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:24:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:24:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:25:52.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:25:52.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:26:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:26:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-22T16:27:52.000Z warn core-service/main.go:78 Failed periodic DB Ping (TotalConns=0) {"Database": "scd"} 2024-05-22T16:27:52.000Z warn core-service/main.go:78 Failed periodic DB Ping (TotalConns=0) {"Database": "rid"} 2024-05-22T16:28:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-22T16:28:52.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} DB bootstrap complete; starting core service... Debug Mode: off 2024-05-23T13:14:52.991Z info core-service/main.go:210 build {"address": ":80", "description": {"Time":"2024-05-03.13:05:22","Commit":"72367c9-dirty","Host":"buildkitsandbox"}} 2024-05-23T13:14:52.992Z info core-service/main.go:211 config {"address": ":80", "scd": true} 2024-05-23T13:14:54.990Z info core-service/main.go:319 Starting DSS HTTP server {"address": ":80"} 2024-05-23T13:15:53.295Z info logging/http.go:84 GET / HTTP/1.1 {"address": ":80", "req_dump": "", "resp_dump": "404 page not found\n", "req_headers": {"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,/;q=0.8"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-US,en;q=0.5"],"Connection":["keep-alive"],"Priority":["u=1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"]}, "resp_status_code": 404, "resp_status_text": "Not Found", "peer_address": "172.19.0.1:64340", "start_time": "2024-05-23T13:15:53.295Z", "duration": "88.692µs"} 2024-05-23T13:15:53.313Z info logging/http.go:84 GET /favicon.ico HTTP/1.1 {"address": ":80", "req_dump": "", "resp_dump": "404 page not found\n", "req_headers": {"Accept":["image/avif,image/webp,/"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Accept-Language":["en-US,en;q=0.5"],"Connection":["keep-alive"],"Priority":["u=6"],"Referer":["http://localhost:8082/"],"Sec-Fetch-Dest":["image"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["same-origin"],"User-Agent":["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"]}, "resp_status_code": 404, "resp_status_text": "Not Found", "peer_address": "172.19.0.1:64340", "start_time": "2024-05-23T13:15:53.312Z", "duration": "40.104µs"} 2024-05-23T13:15:54.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-23T13:15:54.000Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"} 2024-05-23T13:16:54.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "scd"} 2024-05-23T13:16:54.001Z info core-service/main.go:80 Successful periodic DB Ping {"Database": "rid"}

tomas@tomas-All-Series:~/monitoring$ docker container logs local_infra-scd_bootstrapper-1 Allowing time for CRDB node to come up... Bootstrapping SCD DB... 2024/05/22 16:06:42 Database scd does not exist; creating now 2024/05/22 16:06:43 Initial scd database schema version is 0.0.0, target is 3.1.0 2024/05/22 16:06:43 Running upto-v1.0.0-create_initial_version.sql to migrate 0.0.0 to 1.0.0 2024/05/22 16:08:27 Running upto-v2.0.0-support_api_1_0_0.sql to migrate 1.0.0 to 2.0.0 2024/05/22 16:09:57 Running upto-v3.0.0-add_inverted_indices.sql to migrate 2.0.0 to 3.0.0 2024/05/22 16:10:58 Running upto-v3.1.0-create_uss_availability.sql to migrate 3.0.0 to 3.1.0 2024/05/22 16:11:00 Final scd version: 3.1.0 SCD DB bootstrapping complete; notifying other containers... SCD DB is already set up; no need to run bootstrapper.

tomas@tomas-All-Series:~/monitoring$ docker container logs local_infra-rid_bootstrapper-1 Allowing time for CRDB node to come up... Bootstrapping RID DB... 2024/05/22 16:06:42 Database rid does not exist; checking for older "defaultdb" database 2024/05/22 16:06:42 Database defaultdb already exists; reading current state 2024/05/22 16:06:43 Initial defaultdb database schema version is 0.0.0, target is 4.0.0 2024/05/22 16:06:43 Running upto-v1.0.0-create_isa_subs_tables.sql to migrate 0.0.0 to 1.0.0 2024/05/22 16:07:30 Running upto-v2.0.0-add_inverted_indices.sql to migrate 1.0.0 to 2.0.0 2024/05/22 16:11:06 Running upto-v3.0.0-no_change.sql to migrate 2.0.0 to 3.0.0 2024/05/22 16:11:06 Running upto-v3.1.0-add_writer_column.sql to migrate 3.0.0 to 3.1.0 2024/05/22 16:11:24 Running upto-v3.1.1-add_index_by_time_subscriptions.sql to migrate 3.1.0 to 3.1.1 2024/05/22 16:11:47 Running upto-v4.0.0-rename_defaultdb_to_rid.sql to migrate 3.1.1 to 4.0.0 2024/05/22 16:11:52 Final rid version: 4.0.0 RID DB bootstrapping complete; notifying other containers... RID DB is already set up; no need to run bootstrapper. tomas@tomas-All-Series:~/monitoring$ `

For the Load Test I have used Running in a Container. Running the first step docker image build . -t interuss/monitoring I got this log: `omas@tomas-All-Series:~/monitoring$ cd monitoring tomas@tomas-All-Series:~/monitoring/monitoring$ docker image build . -t interuss/monitoring [+] Building 8.1s (16/19) docker:desktop-linux => [internal] load build definition from Dockerfile 0.6s => => transferring dockerfile: 1.85kB 0.0s => [internal] load metadata for docker.io/library/python:3.12-slim 3.4s => [internal] load .dockerignore 0.9s => => transferring context: 2B 0.0s => [internal] load build context 0.9s => => transferring context: 2B 0.0s => CANCELED [ 1/15] FROM docker.io/library/python:3.12-slim@sha256:afc13 1.8s => => resolve docker.io/library/python:3.12-slim@sha256:afc139a0a6409424 0.3s => => sha256:afc139a0a640942491ec481ad8dda10f2c5b753f5c9 1.65kB / 1.65kB 0.0s => => sha256:fd3817f3a855f6c2ada16ac9468e5ee93e361005bd2 1.37kB / 1.37kB 0.0s => => sha256:cf001c2f8af7214144935ae5b37c9e626ccf789117c 6.69kB / 6.69kB 0.0s => => sha256:09f376ebb190216b0459f470e71bec7b5dfa611d66bf00 0B / 29.15MB 2.5s => => sha256:276709cbedc1f168290ee408fca2af2aacfeb4f922ddca1 0B / 3.51MB 2.5s => CACHED [ 2/15] RUN apt-get update && apt-get install -y openssl curl 0.0s => CACHED [ 3/15] RUN apt-get update && apt-get install -y libffi-dev li 0.0s => CACHED [ 4/15] RUN apt-get update && apt-get install -y libxml2-dev l 0.0s => CACHED [ 5/15] RUN apt-get update && apt-get install -y pkg-config li 0.0s => CACHED [ 6/15] RUN mkdir -p /app/monitoring 0.0s => ERROR [ 7/15] COPY ./requirements.txt /app/monitoring/requirements.tx 0.0s => CACHED [ 8/15] RUN pip install -r /app/monitoring/requirements.txt 0.0s => CACHED [ 9/15] RUN rm -rf pycache 0.0s => ERROR [10/15] ADD ./interfaces /app/interfaces 0.0s => ERROR [11/15] ADD ./monitoring /app/monitoring 0.0s => ERROR [12/15] COPY ./monitoring/health_check.sh /app/health_check.sh 0.0s

[ 7/15] COPY ./requirements.txt /app/monitoring/requirements.txt:

---

[10/15] ADD ./interfaces /app/interfaces:

---

[11/15] ADD ./monitoring /app/monitoring:

---

[12/15] COPY ./monitoring/health_check.sh /app/health_check.sh:

Dockerfile:34

32 | ADD ./interfaces /app/interfaces 33 | ADD ./monitoring /app/monitoring 34 | >>> COPY ./monitoring/health_check.sh /app/health_check.sh 35 | RUN chmod 766 /app/health_check.sh 36 | WORKDIR /app/monitoring

ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref bcb3b0c1-e64f-484f-b1cd-2ebbb2394375::rzb6ik7jt0cv90q8b52ht78jl: "/monitoring/health_check.sh": not found tomas@tomas-All-Series:~/monitoring/monitoring$ `

Afterwards, trying to run step 2a docker run -e AUTH_SPEC="NoAuth()" -p 8089:8089 interuss/monitoring locust -f loadtest/locust_files/ISA.py gives the following log:

tomas@tomas-All-Series:~/monitoring/monitoring$ docker run -e AUTH_SPEC="NoAuth()" -p 8089:8089 interuss/monitoring locust -f loadtest/locust_files/~ Unable to find image 'interuss/monitoring:latest' locally latest: Pulling from interuss/monitoring 8a1e25ce7c4f: Pull complete 1103112ebfc4: Pull complete b4b80ef7128d: Pull complete cc7f04ac52f8: Pull complete 87b8bf94a2ac: Pull complete 4d0db4761572: Pull complete b31027416c35: Pull complete 935e248df863: Pull complete 092b6de76b54: Pull complete f5b28c8cc179: Pull complete eba6dcf7198a: Pull complete 4f4fb700ef54: Pull complete 431540209cfa: Pull complete c0558d646afb: Pull complete ff5ec1e7460f: Pull complete fe1e86e9b997: Pull complete d580981272c4: Pull complete Digest: sha256:03d9ce50195cee8bf40f8899be9d72fc4cd53f27f8985ad1ab23e5f753e98887 Status: Downloaded newer image for interuss/monitoring:latest Could not find any locustfile! Ensure file ends in '.py' and see --help for available options.

Following step 3a docker run -e AUTH_SPEC="DummyOAuth(http://host.docker.internal:8085/token,uss1)" --network="interop_ecosystem_network" -p 8089:8089 interuss/monitoring locust -f loadtest/locust_files/ISA.py gives me the following:

docker run -e AUTH_SPEC="DummyOAuth(http://host.docker.internal:8085/token,uss1)" --network="interop_ecosystem_network" -p 8089:8089 interuss/monitoring locust -f loadtest/locust_files/ISA.py [2024-05-23 13:34:06,092] 47a9d84d588c/INFO/locust.main: Starting web interface at http://0.0.0.0:8089 (accepting connections from all network interfaces) [2024-05-23 13:34:06,096] 47a9d84d588c/INFO/locust.main: Starting Locust 1.3.2

Navigating to http://localhost:8089/ does bring up locust, however, if I use the example url provided http://local_infra-dss-1:8082/v1/dss/ it gives me a 100% failure rate. I am assuming that either I am using the wrong adress for this or LoadTest is not properly set up.

[BenjaminPelletier]

ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref bcb3b0c1-e64f-484f-b1cd-2ebbb2394375::rzb6ik7jt0cv90q8b52ht78jl: "/monitoring/health_check.sh": not found

Note that this means the previous command failed. There is an exception this time, but in general, a dependent sequence of commands isn't going to work if one of them fails. When that happens, debugging effort needs to be focused on the thing that isn't working rather than moving on to try the next thing since there's generally no expectation that next thing will work when its dependent action failed. Moving on without resolving the first failure will usually just confuse the issue by providing irrelevant information.

For this issue, the method of building the image has changed since this documentation was written and this documentation wasn't updated -- the new way is to run make image from the repo root, and I've made that update in #700.

Unable to find image 'interuss/monitoring:latest' locally latest: Pulling from interuss/monitoring 8a1e25ce7c4f

In this particular case, you were able to proceed anyway because the image name when built locally is the same as the image hosted on Docker Hub, so the image hosted on Docker Hub was automatically downloaded and used even though no local image had been built. The down side of this is that the Docker Hub latest image is older than the current version of the repo.

Could not find any locustfile! Ensure file ends in '.py' and see --help for available options.

Seems like the command in 2a was not copied correctly; the logs you provided indicate you ran:

docker run -e AUTH_SPEC="NoAuth()" -p 8089:8089 interuss/monitoring locust -f loadtest/locust_files/~

(or perhaps without the tilde; it's hard to tell without line breaks in the right places) Note that this is not the command in 2a; it is missing "ISA.py".

But, you don't want to run 2a any way. Note that 3 says "If testing local DSS instance, be sure that the loadtest (monitoring) container has access to the DSS container". The command in 2a does not have access to the DSS container because the Locust container is not connected to the network the DSS container is on; 3a shows how to fix that, so the command you want is 3a or 3b, NOT 2a or 2b. (3a and 3b are alternatives to 2a and 2b)

docker run -e AUTH_SPEC="DummyOAuth(http://host.docker.internal:8085/token,uss1)" --network="interop_ecosystem_network" -p 8089:8089 interuss/monitoring locust -f loadtest/locust_files/ISA.py

This command needs to be updated to reflect the modern local deployment approach; I've done so in #700. host.docker.internal is no longer generally used. For some machines, host.docker.internal needs to be specified as an extra host to docker, and I think Windows might be one of those systems. But that's irrelevant with the updated command in #700 which uses oauth.authority.localutm.

I use the example url provided http://local_infra-dss-1:8082/v1/dss/

This URL needs to be updated for the same reason as above; I've done so in #700.

[TRMir]

Running it according to the updated instructions seems to be working. I am getting connection in Locust and around 20_ failure rate in my first test run so far. After running make image i get :

[+] Building 4431.8s (20/20) FINISHED docker:desktop-linux => [internal] load build definition from Dockerfile 10.8s => => transferring dockerfile: 1.85kB 0.6s => [internal] load metadata for docker.io/library/python:3.12-slim 129.9s => [internal] load .dockerignore 11.8s => => transferring context: 62B 0.0s => [ 1/15] FROM docker.io/library/python:3.12-slim@sha256:afc139a0a640 340.5s => => resolve docker.io/library/python:3.12-slim@sha256:afc139a0a6409424 6.7s => => sha256:afc139a0a640942491ec481ad8dda10f2c5b753f5c9 1.65kB / 1.65kB 0.0s => => sha256:fd3817f3a855f6c2ada16ac9468e5ee93e361005bd2 1.37kB / 1.37kB 0.0s => => sha256:cf001c2f8af7214144935ae5b37c9e626ccf789117c 6.69kB / 6.69kB 0.0s => => sha256:09f376ebb190216b0459f470e71bec7b5dfa611d 29.15MB / 29.15MB 13.3s => => sha256:276709cbedc1f168290ee408fca2af2aacfeb4f922 3.51MB / 3.51MB 18.1s => => sha256:2e133733af76c2a11aee79b7cfc733cc8065bc53 12.00MB / 12.00MB 21.7s => => extracting sha256:09f376ebb190216b0459f470e71bec7b5dfa611d66bf0084 0.9s => => sha256:ded8879d9a790c3944bc22c2bed1606a9a2d4d293f2a32 243B / 243B 25.2s => => extracting sha256:276709cbedc1f168290ee408fca2af2aacfeb4f922ddca12 8.2s => => sha256:3cf9507408dcb24084c3a20ccd068986cc37d1b136 3.05MB / 3.05MB 34.1s => => extracting sha256:2e133733af76c2a11aee79b7cfc733cc8065bc538d74b80 13.3s => => extracting sha256:ded8879d9a790c3944bc22c2bed1606a9a2d4d293f2a3239 0.0s => => extracting sha256:3cf9507408dcb24084c3a20ccd068986cc37d1b13629d357 8.3s => [internal] load build context 12.1s => => transferring context: 7.36MB 6.0s => [ 2/15] RUN apt-get update && apt-get install -y openssl curl libg 2457.8s => [ 3/15] RUN apt-get update && apt-get install -y libffi-dev libssl- 627.0s => [ 4/15] RUN apt-get update && apt-get install -y libxml2-dev libxslt 88.7s => [ 5/15] RUN apt-get update && apt-get install -y pkg-config libhdf5 164.2s => [ 6/15] RUN mkdir -p /app/monitoring 4.3s => [ 7/15] COPY ./requirements.txt /app/monitoring/requirements.txt 3.6s => [ 8/15] RUN pip install -r /app/monitoring/requirements.txt 494.8s => [ 9/15] RUN rm -rf __pycache__ 11.2s => [10/15] ADD ./interfaces /app/interfaces 8.8s => [11/15] ADD ./monitoring /app/monitoring 8.1s => [12/15] COPY ./monitoring/health_check.sh /app/health_check.sh 10.9s => [13/15] RUN chmod 766 /app/health_check.sh 12.3s => [14/15] WORKDIR /app/monitoring 12.2s => [15/15] RUN mkdir -p /app/uss-host-files 5.2s => exporting to image 7.3s => => exporting layers 6.6s => => writing image sha256:0ed268cb3fe3f64366d74bd7be85c8160f6395879c91d 0.1s => => naming to docker.io/interuss/monitoring 0.1s

After that I run docker run -e AUTH_SPEC="DummyOAuth(http://oauth.authority.localutm:8085/token,uss1)" --network="interop_ecosystem_network" -p 8089:8089 interuss/monitoring locust -f loadtest/locust_files/ISA.py

Navigating and using the following these three steps also works: Navigate to http://localhost:8089/ Start new test with number of Users to spawn and the rate to spawn them. For the Host, provide the DSS Core Service endpoint used for testing. An example of such url is: http://dss.uss1.localutm/v1/dss/ in case local environment is setup with make start-locally

I am including the report of the first test run. Do you think there are other things that need to be fixed or does everything seem to be running correctly? Test.zip

[BenjaminPelletier]

You have too many failures and too long of response times. Here's my attempt to reproduce from a cloned copy of this repo:

make down-locally

make start-locally

make image

docker run -e AUTH_SPEC="DummyOAuth(http://oauth.authority.localutm:8085/token,uss1)" --network="interop_ecosystem_network" -p 8089:8089 interuss/monitoring locust -f loadtest/locust_files/ISA.py

At http://localhost:8089:

• Number of total users to simulate: 250
• Spawn rate (users/second): 10
• Host: http://dss.uss1.localutm/v1/dss/

After about 5 minutes, I clicked Stop and downloaded the report: report_1717183329.322549.zip

Zero failures in 147k requests, no more than ~120ms per request.

Of course, these results are heavily dependent on the type of machine hosting everything and the total request rate and profile -- bumping to 1000 total users at a 250 users/s spawn rate got me one 10-second response from PUT, 1.2 seconds 90th percentile on all request types, and 3 failures in 200k requests averaging around ~600 requests/s. The failures were all like Exception('{"message":"ISA 4ef858a5-2adc-4478-9d49-7cff4d932d06 not found (E:47dbfd70-c728-4e37-b469-9e0e0a8644bc)"}\n') -- no timeouts like your errors.

If you still get errors, what happens if you follow the above steps exactly?

[TRMir]

I had a little bit of an issue when I tried to run test again. The infrastructure and crdb node would not come up and exited with code 1. So I decided to restart and after restart, I could not even get to BIOS and none of my OS. So I reinstalled everything on a completely different computer.

So after the first try, I ran make down locally and started everything from scratch and it also runs fine. 0% failure rate in both of them. Attached file is a report from the first one. Report.zip

[BenjaminPelletier]

Failing hardware is never fun to deal with; I'm glad the different computer worked. Does that mean this issue is resolved?

[TRMir]

Yes, it seems that everything has been fixed. I restarted the pc, downed the local infrastructure, brought it back up and everything works as it should. Thank you very much for your help!

[BenjaminPelletier]

No problem, glad we could get it working! Just let us know if there are any other issues :) Also, just FYI, for smaller or more informal questions, Slack is another good tool. But, issues like this are still good for things that need a lot of documentation support (like logs, etc) like this one.