search

pynbody / tangos

The Agile Numerical Galaxy Organisation System
BSD 3-Clause "New" or "Revised" License
19 stars 13 forks source link

Support for AHF/Ramses importing #178

Closed cphyc closed 2 years ago

cphyc commented 2 years ago

This should follow https://github.com/pynbody/pynbody/pull/661 to add support for AHF/RAMSES in tangos.

I have tested that

Martin-Rey commented 2 years ago

Hi @cphyc @apontzen, a tangential issue to this PR, but the tests are systematically failing in the mysql build for all python versions with

.EEEEEEEEEEEEEE.EEE
[74](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:74)
======================================================================
[75](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:75)
ERROR: test suite for <module 'test_ahf_trees' from '/home/runner/work/tangos/tangos/tests/test_ahf_trees.py'>
[76](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:76)
----------------------------------------------------------------------
[1559](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1559)
Traceback (most recent call last):
[1560](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1560)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/nose/suite.py", line 210, in run
[1561](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1561)
    self.setUp()
[1562](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1562)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/nose/suite.py", line 293, in setUp
[1563](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1563)
    self.setupContext(ancestor)
[1564](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1564)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/nose/suite.py", line 316, in setupContext
[1565](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1565)
    try_run(context, names)
[1566](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1566)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/nose/util.py", line 471, in try_run
[1567](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1567)
    return func()
[1568](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1568)
  File "/home/runner/work/tangos/tangos/tests/test_merger_tree.py", line 16, in setup
[1569](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1569)
    testing.init_blank_db_for_testing()
[1570](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1570)
  File "/home/runner/work/tangos/tangos/tangos/testing/__init__.py", line 176, in init_blank_db_for_testing
[1571](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1571)
    with engine.connect() as conn:
[1572](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1572)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3234, in connect
[1573](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1573)
    return self._connection_cls(self, close_with_result=close_with_result)
[1574](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1574)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 96, in __init__
[1575](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1575)
    else engine.raw_connection()
[1576](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1576)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3313, in raw_connection
[1577](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1577)
    return self._wrap_pool_connect(self.pool.connect, _connection)
[1578](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1578)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 3280, in _wrap_pool_connect
[1579](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1579)
    return fn()
[1580](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1580)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 310, in connect
[1581](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1581)
    return _ConnectionFairy._checkout(self)
[1582](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1582)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 868, in _checkout
[1583](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1583)
    fairy = _ConnectionRecord.checkout(pool)
[1584](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1584)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 476, in checkout
[1585](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1585)
    rec = pool._do_get()
[1586](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1586)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 146, in _do_get
[1587](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1587)
    self._dec_overflow()
[1588](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1588)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
[1589](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1589)
    with_traceback=exc_tb,
[1590](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1590)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
[1591](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1591)
    raise exception
[1592](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1592)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 143, in _do_get
[1593](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1593)
    return self._create_connection()
[1594](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1594)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 256, in _create_connection
[1595](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1595)
    return _ConnectionRecord(self)
[1596](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1596)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 371, in __init__
[1597](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1597)
    self.__connect()
[1598](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1598)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 666, in __connect
[1599](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1599)
    pool.logger.debug("Error on connect(): %s", e)
[1600](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1600)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 72, in __exit__
[1601](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1601)
    with_traceback=exc_tb,
[1602](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1602)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
[1603](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1603)
    raise exception
[1604](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1604)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
[1605](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1605)
    self.dbapi_connection = connection = pool._invoke_creator(self)
[1606](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1606)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/engine/create.py", line 590, in connect
[1607](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1607)
    return dialect.connect(*cargs, **cparams)
[1608](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1608)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 597, in connect
[1609](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1609)
    return self.dbapi.connect(*cargs, **cparams)
[1610](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1610)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/pymysql/connections.py", line 353, in __init__
[1611](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1611)
    self.connect()
[1612](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1612)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/pymysql/connections.py", line 633, in connect
[1613](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1613)
    self._request_authentication()
[1614](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1614)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/pymysql/connections.py", line 932, in _request_authentication
[1615](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1615)
    auth_packet = _auth.caching_sha2_password_auth(self, auth_packet)
[1616](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1616)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/pymysql/_auth.py", line 265, in caching_sha2_password_auth
[1617](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1617)
    data = sha2_rsa_encrypt(conn.password, conn.salt, conn.server_public_key)
[1618](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1618)
  File "/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/pymysql/_auth.py", line 144, in sha2_rsa_encrypt
[1619](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1619)
    "'cryptography' package is required for sha256_password or caching_sha2_password auth methods"
[1620](https://github.com/cphyc/tangos/runs/6020693936?check_suite_focus=true#step:10:1620)
RuntimeError: 'cryptography' package is required for sha256_password or caching_sha2_password auth methods

I am not entirely sure whether this is because the branch is based against an older version of main and needs rebasing, whether it should be fixed here, or through a separate branch (I would expect the current set of tests on main to fail with the same compatibility error?)

apontzen commented 2 years ago

This seems to be a breaking change in PyMySQL or a related package. It is fixed in the main branch and so I would recommend pulling main into here.