Evaluate possible use of diagrams package (replace mermaid) - Githubissues

esnet / sense-rtmon

Custom Scripts for Dynamic End-To-End Flow-Specific Grafana Dashboards

5 stars 4 forks source link

Evaluate possible use of diagrams package (replace mermaid) #127

Open juztas opened 4 months ago

juztas commented 4 months ago

Mermaid is nice, but there is complexity on topology generation and overlapping items (in case big topo). There is diagrams py package - which can generate image and we could show it. Something to try out and see if that gives nicer output. https://diagrams.mingrammer.com/docs/getting-started/examples

juztas commented 3 months ago

Example run uploaded here: https://github.com/esnet/sense-rtmon/pull/130

juztas commented 3 months ago

Here is what needs to be implemented:

[ ] In addition to mermaid - it should also generate diagrams graph and store image on a directory, e.g. /srv/images/
[ ] http endpoint should allow unauthenticated access to get images (will be needed for next step)
[ ] grafana html panel template - which loads specified image in grafana dashboard

We would have it running in parallel for few shots and compare how it looks between mermaid and diagrams

sunami09 commented 1 month ago

manifest = {
    "Ports": [
      {
        "Site": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc502",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_UCSD_OASIS:oasis",
        "Peer": "urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel502",
        "Vlan": "3603",
        "Mac": "?port_mac?",
        "Name": "Pc502"
      },
      {
        "Site": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc500",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_UCSD_OASIS:oasis",
        "Peer": "urn:ogf:network:nrp-nautilus.io:2020:edgecore_s0:PortChannel500",
        "Vlan": "3603",
        "Mac": "?port_mac?",
        "Name": "Pc500"
      },
      {
        "Site": "urn:ogf:network:nrp-nautilus.io:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:nrp-nautilus.io:2020:edgecore_s0:Ethernet32",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_SDSC:edgecore_s0",
        "Peer": "?peer?",
        "Host": [
          {
            "IPv6": "?ipv6?",
            "IPv4": "10.251.87.82/30",
            "Interface": "enp168s0np0",
            "Mac": "a0:88:c2:86:ee:7c",
            "Name": "T2_US_SDSC:k8s-gen5-01.sdsc.optiputer.net"
          }
        ],
        "Vlan": "3603",
        "Mac": "00:90:fb:76:e4:7b",
        "Name": "Ethernet32"
      },
      {
        "Site": "urn:ogf:network:nrp-nautilus.io:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:nrp-nautilus.io:2020:edgecore_s0:PortChannel500",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_SDSC:edgecore_s0",
        "Peer": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc500",
        "Vlan": "3603",
        "Mac": "00:90:fb:76:e4:7b",
        "Name": "PortChannel500"
      },
      {
        "Site": "urn:ogf:network:sc-test.cenic.net:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel501",
        "IPv4": "?port_ipv4?",
        "Node": "NRM_CENIC:aristaeos_s0",
        "Peer": "urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_102",
        "Vlan": "3603",
        "Mac": "28:e7:1d:3f:53:88",
        "Name": "Port-Channel501"
      },
      {
        "Site": "urn:ogf:network:sc-test.cenic.net:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel502",
        "IPv4": "?port_ipv4?",
        "Node": "NRM_CENIC:aristaeos_s0",
        "Peer": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc502",
        "Vlan": "3603",
        "Mac": "28:e7:1d:3f:53:70",
        "Name": "Port-Channel502"
      },
      {
        "Site": "urn:ogf:network:tier2.ultralight.org:2024",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_101",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_Caltech:dellos10_s0",
        "Peer": "urn:ogf:network:ultralight.org:2013:dellos9_s0:Port-channel_103",
        "Vlan": "3603",
        "Mac": "8c:04:ba:e9:0e:a8",
        "Name": "Port-channel 101"
      },
      {
        "Site": "urn:ogf:network:tier2.ultralight.org:2024",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_102",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_Caltech:dellos10_s0",
        "Peer": "urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel501",
        "Vlan": "3603",
        "Mac": "8c:04:ba:e9:0e:a9",
        "Name": "Port-channel 102"
      },
      {
        "Site": "urn:ogf:network:ultralight.org:2013",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:ultralight.org:2013:dellos9_s0:hundredGigE_1-32",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_Caltech_Test:dellos9_s0",
        "Peer": "?peer?",
        "Host": [
          {
            "IPv6": "?ipv6?",
            "IPv4": "10.251.87.81/30",
            "Interface": "mlx5p1s1",
            "Mac": "ec:0d:9a:c1:ba:60",
            "Name": "T2_US_Caltech_Test:sandie-7.ultralight.org"
          }
        ],
        "Vlan": "3603",
        "Mac": "4c:76:25:e8:44:c2",
        "Name": "hundredGigE 1/32"
      },
      {
        "Site": "urn:ogf:network:ultralight.org:2013",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:ultralight.org:2013:dellos9_s0:Port-channel_103",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_Caltech_Test:dellos9_s0",
        "Peer": "urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_101",
        "Vlan": "3603",
        "Mac": "4c:76:25:e8:44:c2",
        "Name": "Port-channel 103"
      }
    ]
  }

sunami09 commented 1 month ago

{
    "Ports": [
      {
        "Site": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc502",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_UCSD_OASIS:oasis",
        "Peer": "urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel502",
        "Vlan": "3603",
        "Mac": "?port_mac?",
        "Name": "Pc502"
      },
      {
        "Site": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc500",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_UCSD_OASIS:oasis",
        "Peer": "urn:ogf:network:nrp-nautilus.io:2020:edgecore_s0:PortChannel500",
        "Vlan": "3603",
        "Mac": "?port_mac?",
        "Name": "Pc500"
      },
      {
        "Site": "urn:ogf:network:nrp-nautilus.io:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:nrp-nautilus.io:2020:edgecore_s0:Ethernet32",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_SDSC:edgecore_s0",
        "Peer": "?peer?",
        "Host": [
          {
            "IPv6": "?ipv6?",
            "IPv4": "10.251.87.82/30",
            "Interface": "enp168s0np0",
            "Mac": "a0:88:c2:86:ee:7c",
            "Name": "T2_US_SDSC:k8s-gen5-01.sdsc.optiputer.net"
          }
        ],
        "Vlan": "3603",
        "Mac": "00:90:fb:76:e4:7b",
        "Name": "Ethernet32"
      },
      {
        "Site": "urn:ogf:network:nrp-nautilus.io:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:nrp-nautilus.io:2020:edgecore_s0:PortChannel500",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_SDSC:edgecore_s0",
        "Peer": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc500",
        "Vlan": "3603",
        "Mac": "00:90:fb:76:e4:7b",
        "Name": "PortChannel500"
      },
      {
        "Site": "urn:ogf:network:sc-test.cenic.net:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel501",
        "IPv4": "?port_ipv4?",
        "Node": "NRM_CENIC:aristaeos_s0",
        "Peer": "urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_102",
        "Vlan": "3603",
        "Mac": "28:e7:1d:3f:53:88",
        "Name": "Port-Channel501"
      },
      {
        "Site": "urn:ogf:network:sc-test.cenic.net:2020",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel502",
        "IPv4": "?port_ipv4?",
        "Node": "NRM_CENIC:aristaeos_s0",
        "Peer": "urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc502",
        "Vlan": "3603",
        "Mac": "28:e7:1d:3f:53:70",
        "Name": "Port-Channel502"
      },
      {
        "Site": "urn:ogf:network:tier2.ultralight.org:2024",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_101",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_Caltech:dellos10_s0",
        "Peer": "urn:ogf:network:ultralight.org:2013:dellos9_s0:Port-channel_103",
        "Vlan": "3603",
        "Mac": "8c:04:ba:e9:0e:a8",
        "Name": "Port-channel 101"
      },
      {
        "Site": "urn:ogf:network:tier2.ultralight.org:2024",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_102",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_Caltech:dellos10_s0",
        "Peer": "urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel501",
        "Vlan": "3603",
        "Mac": "8c:04:ba:e9:0e:a9",
        "Name": "Port-channel 102"
      },
      {
        "Site": "urn:ogf:network:ultralight.org:2013",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:ultralight.org:2013:dellos9_s0:hundredGigE_1-32",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_Caltech_Test:dellos9_s0",
        "Peer": "?peer?",
        "Host": [
          {
            "IPv6": "?ipv6?",
            "IPv4": "10.251.87.81/30",
            "Interface": "mlx5p1s1",
            "Mac": "ec:0d:9a:c1:ba:60",
            "Name": "T2_US_Caltech_Test:sandie-7.ultralight.org"
          }
        ],
        "Vlan": "3603",
        "Mac": "4c:76:25:e8:44:c2",
        "Name": "hundredGigE 1/32"
      },
      {
        "Site": "urn:ogf:network:ultralight.org:2013",
        "IPv6": "?port_ipv6?",
        "Port": "urn:ogf:network:ultralight.org:2013:dellos9_s0:Port-channel_103",
        "IPv4": "?port_ipv4?",
        "Node": "T2_US_Caltech_Test:dellos9_s0",
        "Peer": "urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_101",
        "Vlan": "3603",
        "Mac": "4c:76:25:e8:44:c2",
        "Name": "Port-channel 103"
      }
    ]
  }

sunami09 commented 1 month ago

For Manifest:

{'Ports': [{'Host': [{'IPv4': '10.251.86.138/30',
                      'IPv6': '?ipv6?',
                      'Interface': 'enp168s0np0',
                      'Mac': 'a0:88:c2:86:ee:7c',
                      'Name': 'T2_US_SDSC:k8s-gen5-01.sdsc.optiputer.net'}],
            'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '00:90:fb:76:e4:7b',
            'Name': 'Ethernet32',
            'Node': 'T2_US_SDSC:edgecore_s0',
            'Peer': '?peer?',
            'Port': 'urn:ogf:network:nrp-nautilus.io:2020:edgecore_s0:Ethernet32',
            'Site': 'urn:ogf:network:nrp-nautilus.io:2020',
            'Vlan': '3607'},
           {'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '00:90:fb:76:e4:7b',
            'Name': 'PortChannel500',
            'Node': 'T2_US_SDSC:edgecore_s0',
            'Peer': 'urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc500',
            'Port': 'urn:ogf:network:nrp-nautilus.io:2020:edgecore_s0:PortChannel500',
            'Site': 'urn:ogf:network:nrp-nautilus.io:2020',
            'Vlan': '3607'},
           {'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '4c:76:25:e8:44:c2',
            'Name': 'Port-channel 103',
            'Node': 'T2_US_Caltech_Test:dellos9_s0',
            'Peer': 'urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_101',
            'Port': 'urn:ogf:network:ultralight.org:2013:dellos9_s0:Port-channel_103',
            'Site': 'urn:ogf:network:ultralight.org:2013',
            'Vlan': '3607'},
           {'Host': [{'IPv4': '10.251.86.137/30',
                      'IPv6': '?ipv6?',
                      'Interface': 'mlx5p1s1',
                      'Mac': 'ec:0d:9a:c1:ba:60',
                      'Name': 'T2_US_Caltech_Test:sandie-7.ultralight.org'}],
            'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '4c:76:25:e8:44:c2',
            'Name': 'hundredGigE 1/32',
            'Node': 'T2_US_Caltech_Test:dellos9_s0',
            'Peer': '?peer?',
            'Port': 'urn:ogf:network:ultralight.org:2013:dellos9_s0:hundredGigE_1-32',
            'Site': 'urn:ogf:network:ultralight.org:2013',
            'Vlan': '3607'},
           {'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '28:e7:1d:3f:53:70',
            'Name': 'Port-Channel502',
            'Node': 'NRM_CENIC:aristaeos_s0',
            'Peer': 'urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc502',
            'Port': 'urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel502',
            'Site': 'urn:ogf:network:sc-test.cenic.net:2020',
            'Vlan': '3607'},
           {'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '28:e7:1d:3f:53:88',
            'Name': 'Port-Channel501',
            'Node': 'NRM_CENIC:aristaeos_s0',
            'Peer': 'urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_102',
            'Port': 'urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel501',
            'Site': 'urn:ogf:network:sc-test.cenic.net:2020',
            'Vlan': '3607'},
           {'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '8c:04:ba:e9:0e:a8',
            'Name': 'Port-channel 101',
            'Node': 'T2_US_Caltech:dellos10_s0',
            'Peer': 'urn:ogf:network:ultralight.org:2013:dellos9_s0:Port-channel_103',
            'Port': 'urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_101',
            'Site': 'urn:ogf:network:tier2.ultralight.org:2024',
            'Vlan': '3607'},
           {'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '8c:04:ba:e9:0e:a9',
            'Name': 'Port-channel 102',
            'Node': 'T2_US_Caltech:dellos10_s0',
            'Peer': 'urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel501',
            'Port': 'urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_102',
            'Site': 'urn:ogf:network:tier2.ultralight.org:2024',
            'Vlan': '3607'},
           {'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '?port_mac?',
            'Name': 'Pc500',
            'Node': 'T2_US_UCSD_OASIS:oasis',
            'Peer': 'urn:ogf:network:nrp-nautilus.io:2020:edgecore_s0:PortChannel500',
            'Port': 'urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc500',
            'Site': 'urn:ogf:network:sense-oasis-nrp-nautilus.io:2020',
            'Vlan': '3607'},
           {'IPv4': '?port_ipv4?',
            'IPv6': '?port_ipv6?',
            'Mac': '?port_mac?',
            'Name': 'Pc502',
            'Node': 'T2_US_UCSD_OASIS:oasis',
            'Peer': 'urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel502',
            'Port': 'urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc502',
            'Site': 'urn:ogf:network:sense-oasis-nrp-nautilus.io:2020',
            'Vlan': '3607'}]}

sunami09 commented 1 month ago

@juztas hey what is this error,

Mon, 28 Oct 2024 03:35:11.132 - RTMonLibs.LogLib - INFO - ================================================================================
Mon, 28 Oct 2024 03:35:11.132 - RTMonLibs.LogLib - INFO - Submit Execution: rtmon-debug-sense-o-dev.es.net-6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee, {'state': 'submitted', 'referenceUUID': '6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee', 'orchestrator': 'sense-o-dev.es.net', 'submission': 'AUTH_KEY'}
Mon, 28 Oct 2024 03:35:11.875 - RTMonLibs.LogLib - INFO - Here is instance for 6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee:
Mon, 28 Oct 2024 03:35:11.877 - RTMonLibs.LogLib - INFO - {'alias': 'RTMON-Caltech-SDSC-v2',
 'archived': False,
 'intents': [{'creation_time': '2024-10-22 16:12:00',
              'id': '7fdaf618-b3e6-4f5b-864e-acaa7e456362',
              'json': {'data': {'connections': [{'bandwidth': {'capacity': '1000',
                                                               'qos_class': 'guaranteedCapped'},
                                                 'ip_address_pool': {'name': 'AutoGOLE-IPv4-Test-Pool',
                                                                     'netmask': '/30'},
                                                 'name': 'Connection 1',
                                                 'terminals': [{'assign_ip': True,
                                                                'uri': 'urn:ogf:network:ultralight.org:2013:sandie-1.ultralight.org',
                                                                'vlan_tag': 'any'},
                                                               {'assign_ip': True,
                                                                'uri': 'urn:ogf:network:nrp-nautilus.io:2020:k8s-gen5-02.sdsc.optiputer.net',
                                                                'vlan_tag': 'any'}]}],
                                'type': 'Multi-Path P2P VLAN'},
                       'options': [],
                       'queries': [],
                       'service': 'dnc',
                       'service_instance_uuid': '6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee',
                       'service_profile_uuid': 'ca660eb3-899c-49bf-8e18-d5c0dcbb9fdf'},
              'provisioned': True,
              'serviceDeltaUUID': '5a89c625-fe31-4f34-80d5-6a94c09762be',
              'serviceInstanceUUID': '6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee'}],
 'lastState': 'COMMITTED',
 'owner': 'sdasgupta@lbl.gov',
 'profileUUID': 'ca660eb3-899c-49bf-8e18-d5c0dcbb9fdf',
 'referenceUUID': '6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee',
 'state': 'CREATE - READY',
 'timestamp': '2024/10/22 16:12:00'}
Traceback (most recent call last):
  File "src/python/RTMon/worker.py", line 271, in <module>
    worker.startwork()
  File "src/python/RTMon/worker.py", line 230, in startwork
    self._startwork()
  File "src/python/RTMon/worker.py", line 253, in _startwork
    self.main()
  File "src/python/RTMon/worker.py", line 217, in main
    self.submit_exe(filename, fout)
  File "src/python/RTMon/worker.py", line 50, in submit_exe
    manifest = self.s_getManifest(instance)
  File "/opt/devrtmon/src/python/RTMonLibs/SenseAPI.py", line 79, in s_getManifest
    response = wApi.manifest_create(dumpJson(template, self.logger))
  File "/usr/local/lib/python3.6/site-packages/sense/common.py", line 21, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sense/client/workflow_combined_api.py", line 680, in manifest_create
    body_xml, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.6/site-packages/sense/common.py", line 21, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sense/client/workflow_combined_api.py", line 735, in instance_si_uuid_manifest_post_with_http_info
    query_params=query_params)
  File "/usr/local/lib/python3.6/site-packages/sense/common.py", line 21, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sense/client/requestwrapper.py", line 145, in request
    f"Returned code {ret.status_code} with error '{error_message}'")
ValueError: Returned code 500 with error 'Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (io.undertow.servlet.spec.ServletInputStreamImpl); line: 1, column: 2]'
[root@docker-desktop devrtmon]#

sunami09 commented 1 month ago

@juztas

T2_US_Caltech_Test:sandie-1.ultralight.org', 'vlan': 'vlan.3601'}], 'Vlan': '3601', 'Mac': '4c:76:25:e8:44:c2', 'Name': 'hundredGigE 1/10'}, {'Site': 'urn:ogf:network:ultralight.org:2013', 'IPv6': '?port_ipv6?', 'Port': 'urn:ogf:network:ultralight.org:2013:dellos9_s0:Port-channel_103', 'IPv4': '?port_ipv4?', 'Node': 'T2_US_Caltech_Test:dellos9_s0', 'Peer': 'urn:ogf:network:tier2.ultralight.org:2024:dellos10_s0:Port-channel_101', 'Vlan': '3601', 'Mac': '4c:76:25:e8:44:c2', 'Name': 'Port-channel 103'}, {'Site': 'urn:ogf:network:sense-oasis-nrp-nautilus.io:2020', 'IPv6': '?port_ipv6?', 'Port': 'urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc502', 'IPv4': '?port_ipv4?', 'Node': 'T2_US_UCSD_OASIS:oasis', 'Peer': 'urn:ogf:network:sc-test.cenic.net:2020:aristaeos_s0:Port-Channel502', 'Vlan': '3601', 'Mac': '?port_mac?', 'Name': 'Pc502'}, {'Site': 'urn:ogf:network:sense-oasis-nrp-nautilus.io:2020', 'IPv6': '?port_ipv6?', 'Port': 'urn:ogf:network:sense-oasis-nrp-nautilus.io:2020:oasis:Pc500', 'IPv4': '?port_ipv4?', 'Node': 'T2_US_UCSD_OASIS:oasis', 'Peer': 'urn:ogf:network:nrp-nautilus.io:2020:edgecore_s0:PortChannel500', 'Vlan': '3601', 'Mac': '?port_mac?', 'Name': 'Pc500'}]}, 'dashbInfo': {'uid': '593b0eb9-f5bf-868b-9ed3-ec76ca995613', 'annotation_panels': [306, 307, 308, 409, 410, 411, 512, 513, 514, 615, 616, 617, 718, 719, 720, 821, 822, 823, 924, 925, 926, 1034, 1035, 1036, 1037, 1038, 1039, 1040]}, 'retries': 0}
Mon, 28 Oct 2024 03:37:19.528 - RTMonLibs.LogLib - INFO - Deleting Dashboard: RTMON-Caltech-SDSC-v2|Flow: 7fdaf618-b3e6-4f5b-864e-acaa7e456362|2024/10/22 16:12:00 (Sunami-Prod)
Traceback (most recent call last):
  File "src/python/RTMon/worker.py", line 271, in <module>
    worker.startwork()
  File "src/python/RTMon/worker.py", line 230, in startwork
    self._startwork()
  File "src/python/RTMon/worker.py", line 253, in _startwork
    self.main()
  File "src/python/RTMon/worker.py", line 219, in main
    self.delete_exe(filename, fout)
  File "src/python/RTMon/worker.py", line 104, in delete_exe
    self.g_deleteDashboard(dashbName)
TypeError: g_deleteDashboard() missing 1 required positional argument: 'folderTitle'

https://github.com/esnet/sense-rtmon/blob/96b33b3f3a7d493e9b456f817edb8a5a027c70b3/autogole-api/src/python/RTMon/worker.py#L98

i think it need 2 arguments:

definition: https://github.com/esnet/sense-rtmon/blob/96b33b3f3a7d493e9b456f817edb8a5a027c70b3/autogole-api/src/python/RTMonLibs/GrafanaAPI.py#L87

juztas commented 1 month ago

@juztas hey what is this error,

Mon, 28 Oct 2024 03:35:11.132 - RTMonLibs.LogLib - INFO - ================================================================================
Mon, 28 Oct 2024 03:35:11.132 - RTMonLibs.LogLib - INFO - Submit Execution: rtmon-debug-sense-o-dev.es.net-6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee, {'state': 'submitted', 'referenceUUID': '6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee', 'orchestrator': 'sense-o-dev.es.net', 'submission': 'AUTH_KEY'}
Mon, 28 Oct 2024 03:35:11.875 - RTMonLibs.LogLib - INFO - Here is instance for 6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee:
Mon, 28 Oct 2024 03:35:11.877 - RTMonLibs.LogLib - INFO - {'alias': 'RTMON-Caltech-SDSC-v2',
 'archived': False,
 'intents': [{'creation_time': '2024-10-22 16:12:00',
              'id': '7fdaf618-b3e6-4f5b-864e-acaa7e456362',
              'json': {'data': {'connections': [{'bandwidth': {'capacity': '1000',
                                                               'qos_class': 'guaranteedCapped'},
                                                 'ip_address_pool': {'name': 'AutoGOLE-IPv4-Test-Pool',
                                                                     'netmask': '/30'},
                                                 'name': 'Connection 1',
                                                 'terminals': [{'assign_ip': True,
                                                                'uri': 'urn:ogf:network:ultralight.org:2013:sandie-1.ultralight.org',
                                                                'vlan_tag': 'any'},
                                                               {'assign_ip': True,
                                                                'uri': 'urn:ogf:network:nrp-nautilus.io:2020:k8s-gen5-02.sdsc.optiputer.net',
                                                                'vlan_tag': 'any'}]}],
                                'type': 'Multi-Path P2P VLAN'},
                       'options': [],
                       'queries': [],
                       'service': 'dnc',
                       'service_instance_uuid': '6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee',
                       'service_profile_uuid': 'ca660eb3-899c-49bf-8e18-d5c0dcbb9fdf'},
              'provisioned': True,
              'serviceDeltaUUID': '5a89c625-fe31-4f34-80d5-6a94c09762be',
              'serviceInstanceUUID': '6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee'}],
 'lastState': 'COMMITTED',
 'owner': 'sdasgupta@lbl.gov',
 'profileUUID': 'ca660eb3-899c-49bf-8e18-d5c0dcbb9fdf',
 'referenceUUID': '6803cb62-8ec0-4b70-a6f0-26a1fd2d19ee',
 'state': 'CREATE - READY',
 'timestamp': '2024/10/22 16:12:00'}
Traceback (most recent call last):
  File "src/python/RTMon/worker.py", line 271, in <module>
    worker.startwork()
  File "src/python/RTMon/worker.py", line 230, in startwork
    self._startwork()
  File "src/python/RTMon/worker.py", line 253, in _startwork
    self.main()
  File "src/python/RTMon/worker.py", line 217, in main
    self.submit_exe(filename, fout)
  File "src/python/RTMon/worker.py", line 50, in submit_exe
    manifest = self.s_getManifest(instance)
  File "/opt/devrtmon/src/python/RTMonLibs/SenseAPI.py", line 79, in s_getManifest
    response = wApi.manifest_create(dumpJson(template, self.logger))
  File "/usr/local/lib/python3.6/site-packages/sense/common.py", line 21, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sense/client/workflow_combined_api.py", line 680, in manifest_create
    body_xml, **kwargs)  # noqa: E501
  File "/usr/local/lib/python3.6/site-packages/sense/common.py", line 21, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sense/client/workflow_combined_api.py", line 735, in instance_si_uuid_manifest_post_with_http_info
    query_params=query_params)
  File "/usr/local/lib/python3.6/site-packages/sense/common.py", line 21, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sense/client/requestwrapper.py", line 145, in request
    f"Returned code {ret.status_code} with error '{error_message}'")
ValueError: Returned code 500 with error 'Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: (io.undertow.servlet.spec.ServletInputStreamImpl); line: 1, column: 2]'
[root@docker-desktop devrtmon]#

New issue created here: https://github.com/sdn-sense/sense-o-py-client/issues/35

sunami09 commented 3 weeks ago

?port_mac?', 'Name': 'Pc502'}]}, 'dashbInfo': {'uid': '9019f789-2c56-254d-bf84-9734b2eac740', 'annotation_panels': [306, 307, 308, 409, 410, 411, 512, 513, 514, 615, 616, 617, 718, 719, 720, 821, 822, 823, 924, 925, 926, 1034, 1035, 1036, 1037, 1038, 1039, 1040]}, 'retries': 0}
Fri, 01 Nov 2024 17:27:32.633 - RTMonLibs.LogLib - INFO - Removed diagram image /srv/images/diagram_0a303274-a293-4af8-88b7-5b010d23a0f7.png
Fri, 01 Nov 2024 17:27:32.634 - RTMonLibs.LogLib - INFO - Deleting Dashboard: RTMON-IT6-Caltech-SDSC|Flow: 5d86903e-6b09-4c61-8cca-4d7e36c9aa1a|2024/10/22 16:17:47 (Sunami-Prod)
Fri, 01 Nov 2024 17:27:32.634 - RTMonLibs.LogLib - INFO - Deleting Dashboard: RTMON-IT6-Caltech-SDSC|Flow: 5d86903e-6b09-4c61-8cca-4d7e36c9aa1a|2024/10/22 16:17:47
Fri, 01 Nov 2024 17:27:32.683 - RTMonLibs.LogLib - ERROR - Failed to delete dashboard RTMON-IT6-Caltech-SDSC|Flow: 5d86903e-6b09-4c61-8cca-4d7e36c9aa1a|2024/10/22 16:17:47: Client Error 404: Dashboard not found
Fri, 01 Nov 2024 17:27:33.735 - RTMonLibs.LogLib - ERROR - Failed to delete dashboard RTMON-IT6-Caltech-SDSC|Flow: 5d86903e-6b09-4c61-8cca-4d7e36c9aa1a|2024/10/22 16:17:47: Client Error 404: Dashboard not found
Fri, 01 Nov 2024 17:27:34.791 - RTMonLibs.LogLib - ERROR - Failed to delete dashboard RTMON-IT6-Caltech-SDSC|Flow: 5d86903e-6b09-4c61-8cca-4d7e36c9aa1a|2024/10/22 16:17:47: Client Error 404: Dashboard not found
Traceback (most recent call last):
  File "src/python/RTMon/worker.py", line 272, in <module>
    worker.startwork()
  File "src/python/RTMon/worker.py", line 231, in startwork
    self._startwork()
  File "src/python/RTMon/worker.py", line 254, in _startwork
    self.main()
  File "src/python/RTMon/worker.py", line 220, in main
    self.delete_exe(filename, fout)
  File "src/python/RTMon/worker.py", line 105, in delete_exe
    self.g_deleteDashboard(dashbName, folderName)
  File "/opt/devrtmon/src/python/RTMonLibs/GrafanaAPI.py", line 100, in g_deleteDashboard
    raise Exception(f"Failed to delete dashboard {title} after 3 retries")
Exception: Failed to delete dashboard RTMON-IT6-Caltech-SDSC|Flow: 5d86903e-6b09-4c61-8cca-4d7e36c9aa1a|2024/10/22 16:17:47 after 3 retries

when I removed acces from sense-rtmon

juztas commented 3 weeks ago

It is just a warning - and once deleted it will not complain. Given the fact that there might be multiple rtmon's running (and also you might have supervisor service running) - supervisor deleted it already - but your script started later and tried to do same. Important is to check if dashboard is deleted in grafana (what I can see it is)