search

OpenC3 / cosmos

OpenC3 COSMOS
https://openc3.com
Other
111 stars 31 forks source link

Commanding stops working in 5.19 #1698

Closed tsjordan-eng closed 2 weeks ago

tsjordan-eng commented 2 weeks ago

First check out our main documentation site at https://openc3.com.

Describe the bug After updated from 5.18 to 5.19 I can't command my targets anymore. See logs below.

I tried upgrading my cosmos-project. I stepped from 17, to 18, to 19, to 20, ensuring I kept receiving telemetry along the way. However, I realized at the end that commanding had stopped working so I started with a fresh project with no plugins. I tried installing my plugin that worked on 5.17 on 5.20, then 5.19, then it finally worked again on 5.18.

To Reproduce

I tried installing my plugin that worked on 5.17 on 5.20, then 5.19, then it finally worked again on 5.18

Using Serial or tcpip host interfaces, send a command. Map interface to serial, then try gain, you'll get more errors

logs Command and Telemetry Server:

<TARGET_NAME> System.instance parameters are required on first call
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/system/system.rb:131:in `instance'
(eval):1:in `telemetry'
interface_microservice.rb:650:in `handle_packet'
interface_microservice.rb:586:in `run'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/microservices/microservice.rb:55:in `run'
interface_microservice.rb:811:in `<main>'

I'll map the target to the serial interface and then get the following error whenever I try to command:

Telemetry Viewer

Yellow Banner

Network error: HTTP 500 - Internal Server Error {"jsonrpc":"2.0","method":"cmd","params":"DRIFTER2FLIGHT9 NOOP","id":2,"keyword_params":{"scope":"DEFAULT"}} wrong number of arguments (given 2, expected 1)

Command Sender

Popup window image

cmd-tlm-api container logs

RuntimeError : wrong number of arguments (given 2, expected 1)
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/topics/command_topic.rb:65:in `block in send_command'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/utilities/store_autoload.rb:184:in `block (3 levels) in read_topics'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/utilities/store_autoload.rb:182:in `each'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/utilities/store_autoload.rb:182:in `block (2 levels) in read_topics'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/utilities/store_autoload.rb:181:in `each'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/utilities/store_autoload.rb:181:in `block in read_topics'
/usr/lib/ruby/gems/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:110:in `block (2 levels) in with'
/usr/lib/ruby/gems/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `handle_interrupt'
/usr/lib/ruby/gems/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:109:in `block in with'
/usr/lib/ruby/gems/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `handle_interrupt'
/usr/lib/ruby/gems/3.2.0/gems/connection_pool-2.4.1/lib/connection_pool.rb:106:in `with'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/utilities/store_autoload.rb:61:in `with'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/utilities/store_autoload.rb:177:in `read_topics'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/utilities/store_autoload.rb:89:in `public_send'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/utilities/store_autoload.rb:89:in `method_missing'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/topics/topic.rb:29:in `public_send'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/topics/topic.rb:29:in `method_missing'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/topics/command_topic.rb:55:in `send_command'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/api/cmd_api.rb:542:in `_cmd_implementation'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/api/cmd_api.rb:73:in `cmd'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/io/json_drb.rb:266:in `public_send'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/io/json_drb.rb:266:in `process_request'
/src/app/controllers/api_controller.rb:96:in `handle_post'
/src/app/controllers/api_controller.rb:45:in `block in api'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.20.0/lib/openc3/utilities/open_telemetry.rb:52:in `in_span'
/src/app/controllers/api_controller.rb:31:in `api'

Environment (please complete the following information):

Now that I've upgrade to 20, I can't seem to back down to 18. I get the following error in cmd/tlm server.

Unexpected process died... respawning! ruby multi_microservice.rb DEFAULT__OPENC3__LOG DEFAULT__COMMANDLOG__UNKNOWN DEFAULT__PACKETLOG__UNKNOWN DEFAULT__PERIODIC__DEFAULT DEFAULT__SCOPECLEANUP__DEFAULT DEFAULT__CRITICALCMD__DEFAULT Stdout: {"time":1731228518624551215,"@timestamp":"2024-11-10T08:48:38.624551Z","level":"INFO","microservice_name":"DEFAULT__SCOPEMULTI__DEFAULT","container_name":"7f81a9873074","message":"Microservice initialized with config: {\"name\"=>\"DEFAULT__SCOPEMULTI__DEFAULT\", \"folder_name\"=>nil, \"cmd\"=>[\"ruby\", \"multi_microservice.rb\", \"DEFAULT__OPENC3__LOG\", \"DEFAULT__COMMANDLOG__UNKNOWN\", \"DEFAULT__PACKETLOG__UNKNOWN\", \"DEFAULT__PERIODIC__DEFAULT\", \"DEFAULT__SCOPECLEANUP__DEFAULT\", \"DEFAULT__CRITICALCMD__DEFAULT\"], \"work_dir\"=>\"/openc3/lib/openc3/microservices\", \"ports\"=>[], \"env\"=>{}, \"topics\"=>[], \"target_names\"=>[], \"options\"=>[], \"parent\"=>nil, \"container\"=>nil, \"updated_at\"=>1731219599230683067, \"plugin\"=>nil, \"needs_dependencies\"=>false, \"secrets\"=>[], \"prefix\"=>nil, \"disable_erb\"=>nil, \"ignore_changes\"=>nil}","type":"log"} {"time":1731228521028507722,"@timestamp":"2024-11-10T08:48:41.028507Z","level":"INFO","microservice_name":"DEFAULT__SCOPEMULTI__DEFAULT","container_name":"7f81a9873074","message":"Microservice initialized with config: {\"name\"=>\"DEFAULT__SCOPEMULTI__DEFAULT\", \"folder_name\"=>nil, \"cmd\"=>[\"ruby\", \"multi_microservice.rb\", \"DEFAULT__OPENC3__LOG\", \"DEFAULT__COMMANDLOG__UNKNOWN\", \"DEFAULT__PACKETLOG__UNKNOWN\", \"DEFAULT__PERIODIC__DEFAULT\", \"DEFAULT__SCOPECLEANUP__DEFAULT\", \"DEFAULT__CRITICALCMD__DEFAULT\"], \"work_dir\"=>\"/openc3/lib/openc3/microservices\", \"ports\"=>[], \"env\"=>{}, \"topics\"=>[], \"target_names\"=>[], \"options\"=>[], \"parent\"=>nil, \"container\"=>nil, \"updated_at\"=>1731219599230683067, \"plugin\"=>nil, \"needs_dependencies\"=>false, \"secrets\"=>[], \"prefix\"=>nil, \"disable_erb\"=>nil, \"ignore_changes\"=>nil}","type":"log"} {"time":1731228521862018342,"@timestamp":"2024-11-10T08:48:41.862018Z","level":"INFO","microservice_name":"DEFAULT__OPENC3__LOG","container_name":"7f81a9873074","message":"Microservice initialized with config: {\"name\"=>\"DEFAULT__OPENC3__LOG\", \"folder_name\"=>nil, \"cmd\"=>[\"ruby\", \"text_log_microservice.rb\", \"DEFAULT__OPENC3__LOG\"], \"work_dir\"=>\"/openc3/lib/openc3/microservices\", \"ports\"=>[], \"env\"=>{}, \"topics\"=>[\"DEFAULT__openc3_log_messages\", \"NOSCOPE__openc3_log_messages\"], \"target_names\"=>[], \"options\"=>[[\"CYCLE_TIME\", 600], [\"CYCLE_SIZE\", 50000000]], \"parent\"=>\"DEFAULT__SCOPEMULTI__DEFAULT\", \"container\"=>nil, \"updated_at\"=>1723658045613803310, \"plugin\"=>nil, \"needs_dependencies\"=>false, \"secrets\"=>[], \"prefix\"=>nil, \"disable_erb\"=>nil, \"ignore_changes\"=>nil}","type":"log"} {"time":1731228521918395169,"@timestamp":"2024-11-10T08:48:41.918395Z","level":"INFO","microservice_name":"DEFAULT__PACKETLOG__UNKNOWN","container_name":"7f81a9873074","message":"Microservice initialized with config: {\"name\"=>\"DEFAULT__PACKETLOG__UNKNOWN\", \"folder_name\"=>nil, \"cmd\"=>[\"ruby\", \"log_microservice.rb\", \"DEFAULT__PACKETLOG__UNKNOWN\"], \"work_dir\"=>\"/openc3/lib/openc3/microservices\", \"ports\"=>[], \"env\"=>{}, \"topics\"=>[\"DEFAULT__TELEMETRY__{UNKNOWN}__UNKNOWN\"], \"target_names\"=>[], \"options\"=>[[\"RAW_OR_DECOM\", \"RAW\"], [\"CMD_OR_TLM\", \"TLM\"], [\"CYCLE_TIME\", \"3600\"]], \"parent\"=>\"DEFAULT__SCOPEMULTI__DEFAULT\", \"container\"=>nil, \"updated_at\"=>1723658045620223344, \"plugin\"=>nil, \"needs_dependencies\"=>false, \"secrets\"=>[], \"prefix\"=>nil, \"disable_erb\"=>nil, \"ignore_changes\"=>nil}","type":"log"} {"time":1731228521919066138,"@timestamp":"2024-11-10T08:48:41.919066Z","level":"INFO","microservice_name":"DEFAULT__COMMANDLOG__UNKNOWN","container_name":"7f81a9873074","message":"Microservice initialized with config: {\"name\"=>\"DEFAULT__COMMANDLOG__UNKNOWN\", \"folder_name\"=>nil, \"cmd\"=>[\"ruby\", \"log_microservice.rb\", \"DEFAULT__COMMANDLOG__UNKNOWN\"], \"work_dir\"=>\"/openc3/lib/openc3/microservices\", \"ports\"=>[], \"env\"=>{}, \"topics\"=>[\"DEFAULT__COMMAND__{UNKNOWN}__UNKNOWN\"], \"target_names\"=>[], \"options\"=>[[\"RAW_OR_DECOM\", \"RAW\"], [\"CMD_OR_TLM\", \"CMD\"], [\"CYCLE_TIME\", \"3600\"]], \"parent\"=>\"DEFAULT__SCOPEMULTI__DEFAULT\", \"container\"=>nil, \"updated_at\"=>1723658045618399806, \"plugin\"=>nil, \"needs_dependencies\"=>false, \"secrets\"=>[], \"prefix\"=>nil, \"disable_erb\"=>nil, \"ignore_changes\"=>nil}","type":"log"} {"time":1731228521920522310,"@timestamp":"2024-11-10T08:48:41.920522Z","level":"INFO","microservice_name":"DEFAULT__PERIODIC__DEFAULT","container_name":"7f81a9873074","message":"Microservice initialized with config: {\"name\"=>\"DEFAULT__PERIODIC__DEFAULT\", \"folder_name\"=>nil, \"cmd\"=>[\"ruby\", \"periodic_microservice.rb\", \"DEFAULT__PERIODIC__DEFAULT\"], \"work_dir\"=>\"/openc3/lib/openc3/microservices\", \"ports\"=>[], \"env\"=>{}, \"topics\"=>[], \"target_names\"=>[], \"options\"=>[], \"parent\"=>\"DEFAULT__SCOPEMULTI__DEFAULT\", \"container\"=>nil, \"updated_at\"=>1723658045621790642, \"plugin\"=>nil, \"needs_dependencies\"=>false, \"secrets\"=>[], \"prefix\"=>nil, \"disable_erb\"=>nil, \"ignore_changes\"=>nil}","type":"log"} {"time":1731228522059461658,"@timestamp":"2024-11-10T08:48:42.059461Z","level":"INFO","microservice_name":"DEFAULT__SCOPECLEANUP__DEFAULT","container_name":"7f81a9873074","message":"Microservice initialized with config: {\"name\"=>\"DEFAULT__SCOPECLEANUP__DEFAULT\", \"folder_name\"=>nil, \"cmd\"=>[\"ruby\", \"scope_cleanup_microservice.rb\", \"DEFAULT__SCOPECLEANUP__DEFAULT\"], \"work_dir\"=>\"/openc3/lib/openc3/microservices\", \"ports\"=>[], \"env\"=>{}, \"topics\"=>[], \"target_names\"=>[], \"options\"=>[], \"parent\"=>\"DEFAULT__SCOPEMULTI__DEFAULT\", \"container\"=>nil, \"updated_at\"=>1723658045623366100, \"plugin\"=>nil, \"needs_dependencies\"=>false, \"secrets\"=>[], \"prefix\"=>nil, \"disable_erb\"=>nil, \"ignore_changes\"=>nil}","type":"log"} {"time":1731228522077442256,"@timestamp":"2024-11-10T08:48:42.077442Z","level":"FATAL","microservice_name":"DEFAULT__SCOPEMULTI__DEFAULT","container_name":"7f81a9873074","message":"Microservice DEFAULT__SCOPEMULTI__DEFAULT dying from exception LoadError : cannot load such file -- /openc3/lib/openc3/microservices/critical_cmd_microservice.rb multi_microservice.rb:40:in `require_relative' multi_microservice.rb:40:in `block (3 levels) in run' /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:483:in `set_working_dir_internal' /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:473:in `block in set_working_dir' /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:472:in `synchronize' /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:472:in `set_working_dir' multi_microservice.rb:39:in `block (2 levels) in run'","type":"log"} {"time":1731228522077757000,"@timestamp":"2024-11-10T08:48:42.077757Z","level":"INFO","microservice_name":"DEFAULT__SCOPECLEANUP__DEFAULT","container_name":"7f81a9873074","message":"Shutting down microservice: DEFAULT__SCOPECLEANUP__DEFAULT","type":"log"} {"time":1731228522078248105,"@timestamp":"2024-11-10T08:48:42.078248Z","level":"INFO","microservice_name":"DEFAULT__PERIODIC__DEFAULT","container_name":"7f81a9873074","message":"Shutting down microservice: DEFAULT__PERIODIC__DEFAULT","type":"log"} {"time":1731228522078829515,"@timestamp":"2024-11-10T08:48:42.078829Z","level":"INFO","microservice_name":"DEFAULT__COMMANDLOG__UNKNOWN","container_name":"7f81a9873074","message":"Shutting down microservice: DEFAULT__COMMANDLOG__UNKNOWN","type":"log"} {"time":1731228522079136552,"@timestamp":"2024-11-10T08:48:42.079136Z","level":"INFO","microservice_name":"DEFAULT__PACKETLOG__UNKNOWN","container_name":"7f81a9873074","message":"Shutting down microservice: DEFAULT__PACKETLOG__UNKNOWN","type":"log"} {"time":1731228522199848773,"@timestamp":"2024-11-10T08:48:42.199848Z","level":"INFO","microservice_name":"DEFAULT__OPENC3__LOG","container_name":"7f81a9873074","message":"Shutting down microservice: DEFAULT__OPENC3__LOG","type":"log"} {"time":1731228522200505414,"@timestamp":"2024-11-10T08:48:42.200505Z","level":"INFO","microservice_name":"DEFAULT__SCOPEMULTI__DEFAULT","container_name":"7f81a9873074","message":"Shutting down microservice: DEFAULT__SCOPEMULTI__DEFAULT","type":"log"}  Stderr: #<Thread:0x00007fbaa653cd08 multi_microservice.rb:28 run> terminated with exception (report_on_exception is true): multi_microservice.rb:40:in `require_relative': cannot load such file -- /openc3/lib/openc3/microservices/critical_cmd_microservice.rb (LoadError)    from multi_microservice.rb:40:in `block (3 levels) in run'  from /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:483:in `set_working_dir_internal'  from /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:473:in `block in set_working_dir'  from /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:472:in `synchronize'   from /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:472:in `set_working_dir'   from multi_microservice.rb:39:in `block (2 levels) in run' multi_microservice.rb:40:in `require_relative': cannot load such file -- /openc3/lib/openc3/microservices/critical_cmd_microservice.rb (LoadError)   from multi_microservice.rb:40:in `block (3 levels) in run'  from /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:483:in `set_working_dir_internal'  from /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:473:in `block in set_working_dir'  from /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:472:in `synchronize'   from /usr/lib/ruby/gems/3.2.0/gems/openc3-5.18.0/lib/openc3/top_level.rb:472:in `set_working_dir'   from multi_microservice.rb:39:in `block (2 levels) in run'
ryanmelt commented 2 weeks ago

Please reach out to support@openc3.com and setup a support call. I think this is something specific to your plugin.

hmaclachlan commented 3 days ago

@ryanmelt what did the issue end up being here? We get the same popup window when sending commands to a particular target in recent COSMOS versions

tsjordan-eng commented 3 days ago

Our protocol methods did not pass the extra argument. This was not required before, but is now. Take a look at the Protocol code to see how they pass it through. Once we added it the problem went away.