search

OpenC3 / cosmos

OpenC3 COSMOS
https://openc3.com
Other
111 stars 31 forks source link

Python Conversion Script Is Not Found #1705

Closed lorenzo-gomez-windhover closed 1 week ago

lorenzo-gomez-windhover commented 1 week ago

First check out our main documentation site at https://openc3.com.

Describe the bug When I try to apply a conversion script written in python I get the following error when building the cfs plugin:

Error: Error processing /tmp/d20241113-10-eh3yxi/CFS/cmd_tlm/cfs_tlm.txt:

LoadError : Unable to require double_conversion.py due to cannot load such file -- double_conversion.py. Ensure double_conversion.py is in the OpenC3 lib directory.
<internal:/usr/local/lib/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
<internal:/usr/local/lib/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/top_level.rb:566:in `require'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/top_level.rb:451:in `require_file'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/top_level.rb:439:in `require_class'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/packets/packet_config.rb:565:in `process_current_item'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/packets/packet_config.rb:238:in `block in process_file'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/config/config_parser.rb:217:in `parse_loop'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/config/config_parser.rb:217:in `parse_file'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/packets/packet_config.rb:144:in `process_file'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/system/system.rb:181:in `block in add_target'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/system/system.rb:180:in `each'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/system/system.rb:180:in `add_target'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/system/system.rb:169:in `block in initialize'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/system/system.rb:169:in `each'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/system/system.rb:169:in `initialize'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/models/target_model.rb:602:in `new'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/models/target_model.rb:602:in `deploy'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/models/plugin_model.rb:251:in `block in install_phase2'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/config/config_parser.rb:217:in `parse_loop'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/config/config_parser.rb:217:in `parse_file'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/openc3/models/plugin_model.rb:243:in `install_phase2'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/../bin/openc3cli:314:in `validate_plugin'
/usr/lib/ruby/gems/3.2.0/gems/openc3-5.17.1/lib/../bin/openc3cli:668:in `<main>'

The file targets/CFS/lib/double_conversion.py exists.

To Reproduce Steps to reproduce the behavior:

  1. Generate a conversion script as shown here:https://docs.openc3.com/docs/getting-started/generators#conversion-generator
  2. Generate the new plugin by running openc3.sh cli rake build VERSION=1.29.281

Expected behavior A generated plugin when I generate a python conversion script. This is not an issue when I generate the plugin with a Ruby conversion script.

Screenshots image

Environment (please complete the following information):

Thanks in advance

jmthomas commented 1 week ago

You can only have a target support Ruby or Python. This is specified in the target.txt with LANGUAGE ruby or LANGUAGE python. It looks like you're trying to do both which does not work.

lorenzo-gomez-windhover commented 1 week ago

target.txt

This is specified in the target.txt with LANGUAGE ruby or LANGUAGE python. It looks like you're trying to do both which does not work.

I see the LANGUAGE setting. I set it to python:

LANGUAGE python

When I try the conversion on COSMOS 5.17.1 and 5.20.0, I'm able to generate the plugin, but the telemetry packet is NULL:

image

plugin.txt cfs_tlm.txt Github won't let me push code files as an attachment.

the_great_conversion.py:

from openc3.conversions.conversion import Conversion
class TheGreatConversion(Conversion):
    def __init__(self, multiplier):
        super().__init__()
        self.multiplier = float(multiplier)
    def call(self, value, packet, buffer):
        return value * multiplier
ryanmelt commented 1 week ago

You are using a ruby based Interface in plugin.txt. For python targets the interface must also be in Python

jmthomas commented 1 week ago

The only mix and match with Ruby and Python is scripts run in Script Runner. Everything else in a plugin must choose one language or the other. Open to suggestions on how we can document this better!

lorenzo-gomez-windhover commented 1 week ago

I updated the language and interface to python. I've got the following configuration: plugin.txt:

VARIABLE ip 127.0.0.1
VARIABLE port_tm 1235
VARIABLE port_tc 1234
VARIABLE cfs_target_name CFS

TARGET CFS <%= cfs_target_name %>
# hostname   write_dest_port   read_port   write_src_port   interface_address   ttl   write_timeout   read_timeout   bind_address
INTERFACE <%= cfs_target_name %>_INT udp_interface.py <%= ip %> <%= port_tc %> <%= port_tm %> nil nil 128 nil nil
  MAP_TARGET <%= cfs_target_name %>

target.txt:

LANGUAGE python

# Ignored Parameters
IGNORE_PARAMETER PKTID

# Ignored Items
IGNORE_ITEM PKTID

With this configuration, when I attempt to send a command:

image

Configuration from COSMOS GUI:

image

image

I apologize if I'm missing something obvious...This never happened with the Ruby config.

jmthomas commented 1 week ago

With python you need the full path to the file as shown here: https://docs.openc3.com/docs/configuration/interfaces#udp-interface

INTERFACE <%= cfs_target_name %>_INT openc3/interfaces/udp_interface.py <%= ip %> <%= port_tc %> <%= port_tm %>

Also for Python you should change nil to None but that's just style. It should work either way.

lorenzo-gomez-windhover commented 1 week ago

@jmthomas That was it. Thanks!

Does this mean that the language set by the plugin inside of target.txt is the language that is required for all microservices/interfaces in order for them to work? Which is fine. I just want to make sure I understand fundamentals so I can relay the info to my team.

lorenzo-gomez-windhover commented 1 week ago

While I am able to send commands now after specifying the full path:

VARIABLE ip 127.0.0.1
VARIABLE port_tm 1235
VARIABLE port_tc 1234
VARIABLE cfs_target_name CFS

TARGET CFS <%= cfs_target_name %>
# hostname   write_dest_port   read_port   write_src_port   interface_address   ttl   write_timeout   read_timeout   bind_address
INTERFACE <%= cfs_target_name %>_INT openc3/interfaces/udp_interface.py <%= ip %> <%= port_tc %> <%= port_tm %> None None 128 None None
  MAP_TARGET <%= cfs_target_name %>

target.txt:

LANGUAGE python

# Ignored Parameters
IGNORE_PARAMETER PKTID

# Ignored Items
IGNORE_ITEM PKTID

The CFE_ES_HK_TLM_MID message is still null when I attempt to apply the python conversion:

TELEMETRY CFS CFE_ES_HK_TLM_MID BIG_ENDIAN "Placeholder"
  #                NAME       BITS  TYPE    ID      DESCRIPTION
APPEND_ID_ITEM STREAM_ID 16 UINT 2048 "Stream ID" 
FORMAT_STRING "0x%04X"
APPEND_ITEM SEQUENCE 16 UINT "Packet Sequence" 
FORMAT_STRING "0x%04X"
APPEND_ITEM PKT_LEN 16 UINT "Length of the packet" 
APPEND_ITEM SECONDS 32 UINT "" 
UNITS Seconds sec
APPEND_ITEM SUBSECS 16 UINT "" 
UNITS Milliseconds ms
APPEND_ITEM SPARE2ALIGN 32 UINT "Spares array" 
APPEND_ITEM _Payload_CommandCounter 8 UINT "Command Error Counter:" LITTLE_ENDIAN
READ_CONVERSION the_great_conversion.py 2
APPEND_ITEM _Payload_CommandErrorCounter 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_CFECoreChecksum 16 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_CFEMajorVersion 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_CFEMinorVersion 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_CFERevision 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_CFEMissionRevision 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_OSALMajorVersion 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_OSALMinorVersion 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_OSALRevision 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_OSALMissionRevision 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_PSPMajorVersion 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_PSPMinorVersion 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_PSPRevision 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_PSPMissionRevision 8 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_SysLogBytesUsed 32 UINT "Command Error Counter:" LITTLE_ENDIAN
UNITS Bytes B

APPEND_ITEM _Payload_SysLogSize 32 UINT "Command Error Counter:" LITTLE_ENDIAN
UNITS Bytes B

APPEND_ITEM _Payload_SysLogEntries 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_SysLogMode 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_ERLogIndex 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_ERLogEntries 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_RegisteredCoreApps 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_RegisteredExternalApps 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_RegisteredTasks 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_RegisteredLibs 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_ResetType 32 UINT "Command Error Counter:" LITTLE_ENDIAN
  STATE POWERON 2
STATE PROCESSOR 1

APPEND_ITEM _Payload_ResetSubtype 32 UINT "Command Error Counter:" LITTLE_ENDIAN
  STATE BANKSWITCH_RESET 9
STATE EXCEPTION 6
STATE HWDEBUG_RESET 8
STATE HW_SPECIAL_COMMAND 3
STATE HW_WATCHDOG 4
STATE POWER_CYCLE 1
STATE PUSH_BUTTON 2
STATE RESET_COMMAND 5
STATE SUBTYPE_MAX 10
STATE UNDEFINED_RESET 7

APPEND_ITEM _Payload_ProcessorResets 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_MaxProcessorResets 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_BootSource 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_PerfState 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_PerfMode 32 UINT "Command Error Counter:" LITTLE_ENDIAN
  STATE CENTER 1
STATE END 2
STATE MAX 3
STATE START 0

APPEND_ITEM _Payload_PerfTriggerCount 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ARRAY_ITEM _Payload_PerfFilterMask 32 UINT 128 "" 
FORMAT_STRING 0x%04X

APPEND_ARRAY_ITEM _Payload_PerfTriggerMask 32 UINT 128 "" 
FORMAT_STRING 0x%04X

APPEND_ITEM _Payload_PerfDataStart 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_PerfDataEnd 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_PerfDataCount 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_PerfDataToWrite 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_HeapBytesFree 32 UINT "Command Error Counter:" LITTLE_ENDIAN
UNITS Bytes B

APPEND_ITEM _Payload_HeapBlocksFree 32 UINT "Command Error Counter:" LITTLE_ENDIAN
APPEND_ITEM _Payload_HeapMaxBlockSize 32 UINT "Command Error Counter:" LITTLE_ENDIAN

image

As soon as I remove READ_CONVERSION the_great_conversion.py 2, I am able to see data.

jmthomas commented 1 week ago

Does this mean that the language set by the plugin inside of target.txt is the language that is required for all microservices/interfaces in order for them to work?

Yes, the microservices, interfaces, conversions, limits responses, etc all must be one language. Scripts can still be Ruby or Python but the code points in a target must be consistent.

I'm not sure about your conversion. You can check the logs by doing docker logs cosmos-openc3-operator-1 and see if you notice any stack traces. Also docker logs cosmos-openc3-cosmos-cmd-tlm-api-1.

lorenzo-gomez-windhover commented 1 week ago

Thanks so much for your help (and patience as always). Turns out it was my python script:

from openc3.conversions.conversion import Conversion
class TheGreatConversion(Conversion):
    def __init__(self, multiplier):
        super().__init__()
        self.multiplier = float(multiplier)
    def call(self, value, packet, buffer):
        return value * multiplier

multiplier does not exist since it is part of the class. So I needed to change the script:

from openc3.conversions.conversion import Conversion
class TheGreatConversion(Conversion):
    def __init__(self, multiplier):
        super().__init__()
        self.multiplier = float(multiplier)
    def call(self, value, packet, buffer):
        return value * self.multiplier
jmthomas commented 6 days ago

Hopefully you got a decent stack trace that explained that error? Should've been something in the CmdTlmServer LogMessages. Glad you figured it out.

lorenzo-gomez-windhover commented 4 days ago

Hopefully you got a decent stack trace that explained that error?

Yes I did!

Error from "docker logs -f cosmos-openc3-operator-1"

{"time": 1731944344029601024, "@timestamp": "2024-11-18T15:39:04.029601Z", "level": "ERROR", "microservice_name": "DEFAULTDECOMCFS", "container_name": "e773a7c5650f", "message": "Decom error NameError(\"name 'multiplier' is not defined\")", "type": "log"}

I attempted to build COSMOS from source and it looks like centOS7 is the preferred platform? Any plans to use another distro for development, since as far as I know that distro is not supported maintained(?)

I tried updating the scripts under cosmos/examples/hostinstall/centos7 for an Ubuntu host but failed on Ruby compilation with the following error:

+ mc alias set openc3minio http://localhost:9000 openc3minio openc3miniopassword
INFO[0000] Configuration loaded from file: /etc/traefik/traefik.yaml 
Added `openc3minio` successfully.
+ mc admin policy create openc3minio script /home/lgomez/projects/openc3/cosmos/examples/hostinstall/centos7/../../../openc3-cosmos-init/script-runner.json
<internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require': /openc3/lib/openc3/top_level.rb:468: syntax error, unexpected ')', expecting local variable or method (SyntaxError)
...set_working_dir(working_dir, &)
...                              ^
/openc3/lib/openc3/top_level.rb:470: syntax error, unexpected ')'
...ng_dir_internal(working_dir, &)
...                              ^
/openc3/lib/openc3/top_level.rb:473: syntax error, unexpected ')'
...ng_dir_internal(working_dir, &)
...                              ^
/openc3/lib/openc3/top_level.rb:557: dynamic constant assignment
  Cosmos = OpenC3
  ^~~~~~
/openc3/lib/openc3/top_level.rb:559: module definition in method body
  module CosmosCompatibility
  ^~~~~~~~~~~~~~~~~~~~~~~~~~
/openc3/lib/openc3/top_level.rb:577: class definition in method body
  class Object
  ^~~~~~~~~~~~
/openc3/lib/openc3/top_level.rb:580: syntax error, unexpected end-of-input, expecting `end'
    from <internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /openc3/lib/openc3.rb:43:in `<top (required)>'
    from <internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from <internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from microservice_operator.rb:23:in `<main>'
Created policy `script` successfully.
+ mc admin user add openc3minio scriptrunnerminio scriptrunnerminiopassword
Added user `scriptrunnerminio` successfully.
+ mc admin policy attach openc3minio script --user=scriptrunnerminio
Attached Policies: [script]
To User: scriptrunnerminio
+ mkdir -p /tmp/openc3/tmp/tmp
+ sudo -E --preserve-env=RUBYLIB /openc3/bin/openc3cli load /home/lgomez/projects/openc3/cosmos/examples/hostinstall/centos7/../../../openc3-cosmos-init/plugins/gems/openc3-tool-base-5.17.1.gem
<internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require': /openc3/lib/openc3/top_level.rb:468: syntax error, unexpected ')', expecting local variable or method (SyntaxError)
...set_working_dir(working_dir, &)
...                              ^
/openc3/lib/openc3/top_level.rb:470: syntax error, unexpected ')'
...ng_dir_internal(working_dir, &)
...                              ^
/openc3/lib/openc3/top_level.rb:473: syntax error, unexpected ')'
...ng_dir_internal(working_dir, &)
...                              ^
/openc3/lib/openc3/top_level.rb:557: dynamic constant assignment
  Cosmos = OpenC3
  ^~~~~~
/openc3/lib/openc3/top_level.rb:559: module definition in method body
  module CosmosCompatibility
  ^~~~~~~~~~~~~~~~~~~~~~~~~~
/openc3/lib/openc3/top_level.rb:577: class definition in method body
  class Object
  ^~~~~~~~~~~~
/openc3/lib/openc3/top_level.rb:580: syntax error, unexpected end-of-input, expecting `end'
    from <internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
    from /openc3/lib/openc3.rb:43:in `<top (required)>'
    from **<internal:/usr/local/lib/ruby/site_ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:**
lorenzo-gomez-windhover commented 4 days ago

I realize this is going beyond the scope of this issue. Perhaps I should open up another issue regarding building from source on Ubuntu (?)

ryanmelt commented 4 days ago

The hostinstall example was an old example on how to run COSMOS without using containers. We're going to be removing that from the COSMOS 6 example.

You can build COSMOS from source on any platform by cloning the main https://github.com/openc3/cosmos repo and then running ./openc3.sh build (or ./openc3.bat build on Windows)

lorenzo-gomez-windhover commented 4 days ago

You can build COSMOS from source on any platform by cloning the main https://github.com/openc3/cosmos repo and then running ./openc3.sh build (or ./openc3.bat build on Windows)

I see...this makes sense. Since the docker recipes are copying the files to the image, the dev workflow is:

  1. Add printf
  2. ./openc3.sh stop
  3. ./openc3.sh build
  4. ./openc3.sh run

Am I understanding the dev workflow correctly? I just want to be able to add printf(s) in case I need to when debugging/trying to understand something.

Thanks

ryanmelt commented 4 days ago

You can just do:

  1. Add printf
  2. ./openc3.sh start

That will rebuild any changed containers and stop/start them for you.