Open finchlt opened 2 years ago
You appear to be on the COSMOS 5.0.4 release. This was fixed in COSMOS 5.0.5. Please give that a shot and let us know if your problem is resolved.
Yeah I confirmed we are on 5.0.4. Will try 5.0.5 tomorrow and report back. Thanks for the quick response
Ok so finished with upgrade and migration to 5.0.5 and still getting the same require failure on my custom_protocol.rb in a plugin lib folder.
The new error message for COSMO 5.0.5 is below:
Unexpected process died... respawning! ruby plugin_microservice.rb Stdout: {"time":1657298291946838311,"@timestamp":"2022-07-08T16:38:11.946+00:00","severity":"INFO","microservice_name":"DEFAULT__INTERFACE__MYINT_CMD_TLM_INT","container_name":"be6b3158d10b","log":"Microservice initialized with config:\n{\"name\"=>\"DEFAULT__INTERFACE__MYINT_CMD_TLM_INT\", \"folder_name\"=>nil, \"cmd\"=>[\"ruby\", \"interface_microservice.rb\", \"DEFAULT__INTERFACE__MYINT_CMD_TLM_INT\"], \"work_dir\"=>\"/cosmos/lib/cosmos/microservices\", \"env\"=>{}, \"topics\"=>[], \"target_names\"=>[\"CFE_ES\"], \"options\"=>[], \"container\"=>nil, \"updated_at\"=>1657298197503033280, \"plugin\"=>\"myplugin-1.0.0.20220708163610.gem__20220708163636\", \"needs_dependencies\"=>false}"} {"time":1657298293327031970,"@timestamp":"2022-07-08T16:38:13.327+00:00","severity":"INFO","microservice_name":"DEFAULT__INTERFACE__MYINT_CMD_TLM_INT","container_name":"be6b3158d10b","log":"Microservice initialized with config:\n{\"name\"=>\"DEFAULT__INTERFACE__MYINT_CMD_TLM_INT\", \"folder_name\"=>nil, \"cmd\"=>[\"ruby\", \"interface_microservice.rb\", \"DEFAULT__INTERFACE__MYINT_CMD_TLM_INT\"], \"work_dir\"=>\"/cosmos/lib/cosmos/microservices\", \"env\"=>{}, \"topics\"=>[], \"target_names\"=>[\"CFE_ES\"], \"options\"=>[], \"container\"=>nil, \"updated_at\"=>1657298197503033280, \"plugin\"=>\"myplugin-1.0.0.20220708163610.gem__20220708163636\", \"needs_dependencies\"=>false}"} {"time":1657298293981044600,"@timestamp":"2022-07-08T16:38:13.981+00:00","severity":"INFO","microservice_name":"DEFAULT__INTERFACE__MYINT_CMD_TLM_INT","container_name":"be6b3158d10b","log":"Retrieving DEFAULT/target_archives/CFE_ES/CFE_ES_current.zip from targets bucket"} {"time":1657298294119971718,"@timestamp":"2022-07-08T16:38:14.119+00:00","severity":"ERROR","microservice_name":"DEFAULT__INTERFACE__MYINT_CMD_TLM_INT","container_name":"be6b3158d10b","log":"Unable to require custom_protocol.rb due to cannot load such file -- custom_protocol.rb. Ensure custom_protocol.rb is in the COSMOS lib directory."} Stderr: interface_microservice.rb:520:in `shutdown': undefined method `name' for nil:NilClass (NoMethodError) from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/microservices/microservice.rb:110:in `block in initialize' <internal:/usr/local/lib/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require': Unable to require custom_protocol.rb due to cannot load such file -- custom_protocol.rb. Ensure custom_protocol.rb is in the COSMOS lib directory. (LoadError) from <internal:/usr/local/lib/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/top_level.rb:451:in `require_file' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/top_level.rb:439:in `require_class' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:141:in `block in build' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:140:in `each' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:140:in `build' from interface_microservice.rb:246:in `initialize' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/microservices/microservice.rb:43:in `new' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/microservices/microservice.rb:43:in `run' from interface_microservice.rb:531:in `<main>' <internal:/usr/local/lib/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- custom_protocol.rb (LoadError) from <internal:/usr/local/lib/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/top_level.rb:451:in `require_file' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/top_level.rb:439:in `require_class' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:141:in `block in build' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:140:in `each' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:140:in `build' from interface_microservice.rb:246:in `initialize' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/microservices/microservice.rb:43:in `new' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/microservices/microservice.rb:43:in `run' from interface_microservice.rb:531:in `<main>' <internal:/usr/local/lib/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require': Unable to require custom_protocol.rb due to cannot load such file -- custom_protocol.rb. Ensure custom_protocol.rb is in the COSMOS lib directory. (LoadError) from <internal:/usr/local/lib/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/top_level.rb:451:in `require_file' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/top_level.rb:439:in `require_class' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:141:in `block in build' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:140:in `each' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:140:in `build' from interface_microservice.rb:246:in `initialize' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/microservices/microservice.rb:43:in `new' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/microservices/microservice.rb:43:in `run' from interface_microservice.rb:531:in `<main>' <internal:/usr/local/lib/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- custom_protocol.rb (LoadError) from <internal:/usr/local/lib/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/top_level.rb:451:in `require_file' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/top_level.rb:439:in `require_class' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:141:in `block in build' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:140:in `each' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/models/interface_model.rb:140:in `build' from interface_microservice.rb:246:in `initialize' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/microservices/microservice.rb:43:in `new' from /usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib/cosmos/microservices/microservice.rb:43:in `run' from interface_microservice.rb:531:in `<main>'
I'm going to try adding some additional logging on what the ruby $LOAD_PATH
is at various points of starting my interface to see if I can learn more as well.
Was able to get the $LOAD_PATH
from the point in top_level.rb
where the error is emitted.
The LOAD_PATH interestingly does not include /gems/gem
but only what appears to be part of the temp path created in plugin_model.rb
for the final target loaded See /tmp/d20220708-142-cyu8lx/targets/CFE_ES/procedures
at end of the $LOAD_PATH
$LOAD_PATH:
LOAD_PATH: ["/usr/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib", "/usr/lib/ruby/gems/3.0.0/gems/json-2.6.2/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/json-2.6.2", "/usr/lib/ruby/gems/3.0.0/gems/webrick-1.7.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/yard-0.9.28/lib", "/usr/lib/ruby/gems/3.0.0/gems/uuidtools-2.2.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/snmp-1.3.2/lib", "/usr/lib/ruby/gems/3.0.0/gems/rubyzip-2.3.2/lib", "/usr/lib/ruby/gems/3.0.0/gems/nokogiri-1.13.6-x86_64-linux/lib", "/usr/lib/ruby/gems/3.0.0/gems/nio4r-2.5.8/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/nio4r-2.5.8", "/usr/lib/ruby/gems/3.0.0/gems/puma-5.6.4/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/puma-5.6.4", "/usr/lib/ruby/gems/3.0.0/gems/rack-2.2.3.1/lib", "/usr/lib/ruby/gems/3.0.0/gems/redis-4.7.1/lib", "/usr/lib/ruby/gems/3.0.0/gems/matrix-0.4.2/lib", "/usr/lib/ruby/gems/3.0.0/gems/httpclient-2.8.3/lib", "/usr/lib/ruby/gems/3.0.0/gems/jmespath-1.6.1/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-partitions-1.603.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-eventstream-1.2.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-sigv4-1.5.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.131.2/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-sdk-kms-1.57.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-sdk-s3-1.114.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby", "/usr/lib/ruby/gems/3.0.0/gems/tzinfo-2.0.4/lib", "/usr/lib/ruby/gems/3.0.0/gems/tzinfo-data-1.2022.1/lib", "/usr/lib/ruby/gems/3.0.0/gems/ffi-1.15.5/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/ffi-1.15.5", "/usr/lib/ruby/gems/3.0.0/gems/childprocess-4.1.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/connection_pool-2.2.5/lib", "/usr/lib/ruby/gems/3.0.0/gems/raabro-1.4.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/et-orbi-1.2.7/lib", "/usr/lib/ruby/gems/3.0.0/gems/fugit-1.5.3/lib", "/usr/lib/ruby/gems/3.0.0/gems/rufus-scheduler-3.8.1/lib", "/usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/cosmos-5.0.5", "/usr/lib/ruby/gems/3.0.0/gems/racc-1.6.0/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/racc-1.6.0", "/usr/lib/site_ruby/$(ruby_ver)/x86_64-linux", "/usr/local/lib/site_ruby/3.0.0", "/usr/local/lib/site_ruby/3.0.0/x86_64-linux-musl", "/usr/local/lib/site_ruby", "/usr/lib/ruby/vendor_ruby/3.0.0", "/usr/lib/ruby/vendor_ruby/3.0.0/x86_64-linux-musl", "/usr/lib/ruby/vendor_ruby", "/usr/lib/ruby/3.0.0", "/usr/lib/ruby/3.0.0/x86_64-linux-musl", "/tmp/d20220708-142-cyu8lx/targets/CFE_ES/procedures"]
If you look in your output, it has needs_dependencies=>false. That needs to be true for it to work. Did you uninstall and reinstall the plugin after upgrading?
If you look here: https://github.com/BallAerospace/COSMOS/blob/7e14868c27f51022479fb8ad87d3d1f9c12de35b/cosmos/lib/cosmos/models/plugin_model.rb#L170
This is the line that was changed that sets "needs_dependencies" in COSMOS 5.0.5. But it will require uninstalling and reinstalling the plugin to take effect.
Ok tried this again with similar non-working results.
Additionally I added the following lines to plugin_model.rb
after the current line 170
puts "Plugin #{gem_path} Needs Dependencies: #{needs_dependencies}"
This resulted in the following logged message:
Plugin /tmp/d20220709-197-z178pu/gem Needs Dependencies: true
So it appears the lib
dectection is working at that point in the flow.
However things seem to break down later when the microservices are actually started.
See the interface microservice initialization (from weg gui logs):
Microservice initialized with config: {"name"=>"DEFAULT__INTERFACE__TEST_CMD_TLM_INT", "folder_name"=>nil, "cmd"=>["ruby", "interface_microservice.rb", "DEFAULT__INTERFACE__TEST_CMD_TLM_INT"], "work_dir"=>"/cosmos/lib/cosmos/microservices", "env"=>{}, "topics"=>[], "target_names"=>["CFE_ES"], "options"=>[], "container"=>nil, "updated_at"=>1657382074983207181, "plugin"=>"cosmos-myplugin-sat-1.0.0.gem__20220709155433", "needs_dependencies"=>false}
It really seems like this needs_dependencies flag is not getting persisted and loaded correctly.
Addionally here is the same require error again with additional $LOAD_PATH logging, where there is no added /tmp/d20220709-197-z178pu/gem/lib
as would be expected.
Unable to require custom_protocl.rb due to cannot load such file -- custom_protocl.rb. Ensure custom_protocl.rb is in the COSMOS lib directory. LOAD_PATH: ["/usr/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib", "/usr/lib/ruby/gems/3.0.0/gems/json-2.6.2/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/json-2.6.2", "/usr/lib/ruby/gems/3.0.0/gems/webrick-1.7.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/yard-0.9.28/lib", "/usr/lib/ruby/gems/3.0.0/gems/uuidtools-2.2.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/snmp-1.3.2/lib", "/usr/lib/ruby/gems/3.0.0/gems/rubyzip-2.3.2/lib", "/usr/lib/ruby/gems/3.0.0/gems/nokogiri-1.13.6-x86_64-linux/lib", "/usr/lib/ruby/gems/3.0.0/gems/nio4r-2.5.8/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/nio4r-2.5.8", "/usr/lib/ruby/gems/3.0.0/gems/puma-5.6.4/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/puma-5.6.4", "/usr/lib/ruby/gems/3.0.0/gems/rack-2.2.3.1/lib", "/usr/lib/ruby/gems/3.0.0/gems/redis-4.7.1/lib", "/usr/lib/ruby/gems/3.0.0/gems/matrix-0.4.2/lib", "/usr/lib/ruby/gems/3.0.0/gems/httpclient-2.8.3/lib", "/usr/lib/ruby/gems/3.0.0/gems/jmespath-1.6.1/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-partitions-1.603.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-eventstream-1.2.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-sigv4-1.5.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-sdk-core-3.131.2/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-sdk-kms-1.57.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/aws-sdk-s3-1.114.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby", "/usr/lib/ruby/gems/3.0.0/gems/tzinfo-2.0.4/lib", "/usr/lib/ruby/gems/3.0.0/gems/tzinfo-data-1.2022.1/lib", "/usr/lib/ruby/gems/3.0.0/gems/ffi-1.15.5/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/ffi-1.15.5", "/usr/lib/ruby/gems/3.0.0/gems/childprocess-4.1.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/connection_pool-2.2.5/lib", "/usr/lib/ruby/gems/3.0.0/gems/raabro-1.4.0/lib", "/usr/lib/ruby/gems/3.0.0/gems/et-orbi-1.2.7/lib", "/usr/lib/ruby/gems/3.0.0/gems/fugit-1.5.3/lib", "/usr/lib/ruby/gems/3.0.0/gems/rufus-scheduler-3.8.1/lib", "/usr/lib/ruby/gems/3.0.0/gems/cosmos-5.0.5/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/cosmos-5.0.5", "/usr/lib/ruby/gems/3.0.0/gems/racc-1.6.0/lib", "/usr/lib/ruby/gems/3.0.0/extensions/x86_64-linux-musl/3.0.0/racc-1.6.0", "/usr/lib/site_ruby/$(ruby_ver)/x86_64-linux", "/usr/local/lib/site_ruby/3.0.0", "/usr/local/lib/site_ruby/3.0.0/x86_64-linux-musl", "/usr/local/lib/site_ruby", "/usr/lib/ruby/vendor_ruby/3.0.0", "/usr/lib/ruby/vendor_ruby/3.0.0/x86_64-linux-musl", "/usr/lib/ruby/vendor_ruby", "/usr/lib/ruby/3.0.0", "/usr/lib/ruby/3.0.0/x86_64-linux-musl", "/tmp/d20220709-110-koxj8p/targets/CFE_ES/procedures"]
I also added some additional logging in interface_model.rb with the results below
Added logging initialize funtion of needs_dependencies when interface object is created.
Interface initialize. Needs depdencies: false
Existing print of interface config json:
{"time":1657383671345540456,"@timestamp":"2022-07-09T16:21:11.345+00:00","severity":"INFO","conta
iner_name":"0647dc181602","log":"Configured interface microservice DEFAULT__INTERFACE__TEST_CMD_TLM_INT"}
At interface in deploy function (added at last line logged @needs_dependencies):
Interface deploy. Needs Dependencies: false
In plugin_model.rb after current line 218. logged current_model.needs_dependencies
Current model needs dependencies: false
Additionally this does seem like a really opaque way to handle adding a simple lib path (which in theory should just work for a ruby gem) Additionally this could probably be documented better.
From a user perspective a directive in the plugin.txt mightbe easier to understand with added benefit of being able to turn on/off using a plugin lib folder and initialize everything to the right needs_dependencies
state from the start. However i also understand the desire for this to be transparent to the user.
I think I found the issue. See the following lines in the files below:
These lines are the various calls from class function handle_config from plugin_model.rb
below
This call currently passes the needs_dependencies
input, but the above models did not apply it internally. This results in the models not correctly applying the plugin lib
folder as added below in #1703
I have a pull request ready that applies the fixes to the above models.
It is possible to have a shared lib folder for a plugin. The directory structure is: plugin_root
The plugin gem build is successful as the default file glob
s.files = Dir.glob("{targets,lib,procedures,tools,microservices}/**/*") + %w(Rakefile README.md plugin.txt)
successful grabs and includes my plugin level lib folder.We have a large amount of shared ruby library code that in COSMOS 4.x was just in the the lib folder of the project. This library code is used across many plugins, and I would prefer not to duplicate this to many target specific lib folders.
It appears that this partially working (during plugin initialization). In plugin_model.rb all folders with name "lib" are added to the require search path. This allow imports to resolve at this point and the plugin gemfile to load successfully.
However later when my interfaces is being initialized on I get errors below for a custom protocol
I also get the additional error from the MicroserviceOperator below. I think this is the same issue but thought I would include it as well.