add geant lg agent - Githubissues

Laur04 commented 1 year ago

Added geantlg.py and configuration files to allow for parsing geant's lg
Modified cern lg files to include cern in the title

fno2010 commented 1 year ago

I will suggest you develop new features on a different branch. Keeping the main branch clean is a good practice :)

Some quick comments:

test.py and peering_info.csv are unnecessary. Don't check them in. You can put them into some git-ignored directory (e.g., tmp/)
You introduce a new dependency xmltodict. Make sure you add this dependency to the setup.cfg file.

fno2010 commented 1 year ago

After updating setup.cfg, the docker image needs to be rebuilt. But you can modify the docker-compose.yml to avoid this:

...
  # alto-lg-agent-cern:
  #   image: openalto/alto
  #   volumes:
  #     - ./etc/lg-agent-cern.json:/etc/lg-agent-cern.json
  #   entrypoint: python
  #   command: ["-m", "alto.agent.manage", "--pid", "/tmp", "start", "-c", "/etc/lg-agent-cern.json", "-D", "cernlg"]
  alto-lg-agent-geant:
    image: openalto/alto
    volumes:
      - ./src/alto:/usr/local/lib/python3.10/site-packages/alto
      - ./etc/lg-agent-geant.json:/etc/lg-agent-geant.json
    # entrypoint: python
    # command: ["-m", "alto.agent.manage", "--pid", "/tmp", "start", "-c", "/etc/lg-agent-geant.json", "-D", "geantlg"]
    entrypoint: bash
    command: "-c 'pip install xmltodict && python -m alto.agent.manage --pid /tmp start -c /etc/lg-agent-geant.json -D geantlg'"
    network_mode: "service:alto-frontend"
  alto-cric-agent:
    image: openalto/alto
    volumes:
      - ./src/alto:/usr/local/lib/python3.10/site-packages/alto
      - ./etc/cric-agent.json:/etc/cric-agent.json
    entrypoint: python
    command: ["-m", "alto.agent.manage", "--pid", "/tmp", "start", "-c", "/etc/cric-agent.json", "-D", "cric"]
    network_mode: "service:alto-frontend"
...

You can copy the docker-compose.yml to a new yaml file, e.g., docker-compose-test.yml, and run docker-compose -f docker-compose-test.yml up -d. This can avoid updating the original docker-compose.yml. But don't upload the new yaml file. It is just for your local test.

Right now, lg-agent-cern and lg-agent-geant cannot be started on the same ALTO server because both of them will update the same forwarding db. It will have conflict. We haven't resolved the conflict in the current version. To avoid this, one way is to comment one of them (like the modified docker-compose.yml above); The other way is to modify the agent configuration JSON file and put them into different namespaces.

fno2010 commented 1 year ago

Just dump some error logs for you to debug:

➜ docker-compose logs alto-lg-agent-geant
alto-alto-lg-agent-geant-1  | Collecting xmltodict
alto-alto-lg-agent-geant-1  |   Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)
alto-alto-lg-agent-geant-1  | Installing collected packages: xmltodict
alto-alto-lg-agent-geant-1  | Successfully installed xmltodict-0.13.0
alto-alto-lg-agent-geant-1  | WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system pack
age manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
alto-alto-lg-agent-geant-1  |
alto-alto-lg-agent-geant-1  | [notice] A new release of pip available: 22.2.2 -> 22.3
alto-alto-lg-agent-geant-1  | [notice] To update, run: pip install --upgrade pip
alto-alto-lg-agent-geant-1  | INFO:root:Initializing geantlg Agent...
alto-alto-lg-agent-geant-1  | INFO:root:Loading databases
alto-alto-lg-agent-geant-1  | INFO:root:Starting geantlg Agent...
alto-alto-lg-agent-geant-1  | INFO:root:Loadingroutes on None
alto-alto-lg-agent-geant-1  | DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): lg.geant.org:443
alto-alto-lg-agent-geant-1  | DEBUG:urllib3.connectionpool:https://lg.geant.org:443 "POST /rest/submit HTTP/1.1" 404 0
alto-alto-lg-agent-geant-1  | Traceback (most recent call last):
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 971, in json
alto-alto-lg-agent-geant-1  |     return complexjson.loads(self.text, **kwargs)
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
alto-alto-lg-agent-geant-1  |     return _default_decoder.decode(s)
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
alto-alto-lg-agent-geant-1  |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
alto-alto-lg-agent-geant-1  |     raise JSONDecodeError("Expecting value", s, err.value) from None
alto-alto-lg-agent-geant-1  | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
alto-alto-lg-agent-geant-1  |
alto-alto-lg-agent-geant-1  | During handling of the above exception, another exception occurred:
alto-alto-lg-agent-geant-1  |
alto-alto-lg-agent-geant-1  | Traceback (most recent call last):
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
alto-alto-lg-agent-geant-1  |     return _run_code(code, main_globals, None,
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
alto-alto-lg-agent-geant-1  |     exec(code, run_globals)
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/site-packages/alto/agent/manage.py", line 126, in <module>
alto-alto-lg-agent-geant-1  |     agent.run()
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/site-packages/alto/agent/geantlg.py", line 123, in run
alto-alto-lg-agent-geant-1  |     self.update()
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/site-packages/alto/agent/geantlg.py", line 107, in update
alto-alto-lg-agent-geant-1  |     routes = self.get_routes(_router)
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/site-packages/alto/agent/geantlg.py", line 99, in get_routes
alto-alto-lg-agent-geant-1  |     results_dict = self._do_query(router=router, args=args)
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/site-packages/alto/agent/geantlg.py", line 73, in _do_query
alto-alto-lg-agent-geant-1  |     doctree = data.json()['output'][router]['commandResult']
alto-alto-lg-agent-geant-1  |   File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 975, in json
alto-alto-lg-agent-geant-1  |     raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
alto-alto-lg-agent-geant-1  | requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

openalto / alto

add geant lg agent #25