Problems in ska3 in RedHat 8

taldcroft commented 2 years ago

DS has already baselined RedHat 8.x for ASCDS and syshelp is planning to migrate HEAD linux computers to RH8.

Initial testing on han8-v.

We're moving testing to owen-v to run on an Intel system.

@taldcroft did some initial testing:

git is incompatible with ska3 libraries (like the old emacs incompatibility but more impacting) : Javier

ska3-han8-v$ git pull
/bin/ssh: symbol lookup error: /bin/ssh: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Answer: reinstate git and test.

ska_testr shows many errors

Initial testing showed a lot of errors. I have not dug into any of them. See:

/home/aldcroft/git/ska_testr/outputs/logs/Linux_2022-05-02T10-42-57_2022.3-348b3e2_han8-v.cfa.harvard.edu

Ska.DBI [CLOSED, working now for Tom and Jean]

~All Sybase tests failing~

Ska.Shell : Javier [Closed with /sot/Ska.Shell/issues/25]

Weird.

    def test_env(self):
        envs = getenv('export TEST_ENV_VARA="hello"')
        assert envs['TEST_ENV_VARA'] == 'hello'
        outlines = bash('echo $TEST_ENV_VARA', env=envs)
>       assert outlines == ['hello']
E       AssertionError: assert ['he  o'] == ['hello']
E         At index 0 diff: 'he  o' != 'hello'
E         Full diff:
E         - ['hello']
E         ?     ^^
E         + ['he  o']
E         ?     ^^

Ska.engarchive : Tom / Jean for arc5gl

Arc5gl not working
These ones again, but MAUDE access is working in other tests. Discussed in https://github.com/sot/eng_archive/pull/228.
https://cs.github.com/sot/eng_archive/blob/7e3cfa5b0c7804536526fcf157c45d3643544933/Ska/engarchive/tests/test_fetch.py#L156

Possible solution to expand dt=2000, or else have the fetch just go to NOW.

=========================== short test summary info ============================
FAILED Ska/engarchive/tests/test_fetch.py::test_fetch_derived_param_aliases[sources0-DP_piTch_fss]
FAILED Ska/engarchive/tests/test_fetch.py::test_fetch_derived_param_aliases[sources0-Calc_pitCH_fss]
FAILED Ska/engarchive/tests/test_fetch.py::test_fetch_derived_param_aliases[sources1-DP_piTch_fss]
FAILED Ska/engarchive/tests/test_fetch.py::test_fetch_derived_param_aliases[sources1-Calc_pitCH_fss]
FAILED Ska/engarchive/tests/test_fetch.py::test_fetch_derived_param_aliases[sources2-DP_piTch_fss]
FAILED Ska/engarchive/tests/test_fetch.py::test_fetch_derived_param_aliases[sources2-Calc_pitCH_fss]
============= 6 failed, 93 passed, 1 skipped in 271.18s (0:04:31) ==============

acis_thermal_check [CLOSED, working for John Z and Tom]

Looks like a problem in kadi / django.


_________________ ERROR at setup of test_prediction[MAR0617A] __________________

self = <django.db.backends.utils.CursorDebugWrapper object at 0x7f4c69d9a790> sql = 'SELECT "events_radzone"."start", "events_radzone"."stop", "events_radzone"."tstart", "events_radzone"."tstop", "event...radzone" WHERE ("events_radzone"."start" <= %s AND "events_radzone"."stop" > %s) ORDER BY "events_radzone"."start" ASC' params = ('2017:238:03:30:01.285', '2017:236:23:42:54.816') ignored_wrapper_args = (False, {'connection': <django.db.backends.sqlite3.base.DatabaseWrapper object at 0x7f4c75bcc760>, 'cursor': <django.db.backends.utils.CursorDebugWrapper object at 0x7f4c69d9a790>})

def _execute(self, sql, params, *ignored_wrapper_args):
    self.db.validate_no_broken_transaction()
    with self.db.wrap_database_errors:
        if params is None:
            # params default might be backend specific.
            return self.cursor.execute(sql)
        else:

          return self.cursor.execute(sql, params)

django/db/backends/utils.py:84:



### kadi : Tom [CLOSED with #857]

Not immediately obvious from the stack trace, needs investigation. Most tests passing however, so the `acis_thermal_check` failure got more confusing.

jeanconn commented 2 years ago

It looks like the sybase Ska.DBI tests are passing now for me in ska3 on han8-v, so not sure if the above report was a hiccup.

jeanconn commented 2 years ago

For arc5gl, it looks like there are two issues:

1) We've still got Ska.arc5gl calling /proj/sot/ska/bin/arc5gl which is hardwired to use the old perl. It could use the env perl, but I'm still not sure if we'd also want to put a /proj/sot/ska3/flight/bin/arc5gl wrapper and use it from Ska.arc5gl so that there was no dependence on /proj/sot/ska/bin.

2) There is a tiny issue where the logic to determine which ASCDS release to use is looking for a CentOS in the string. The easiest fix is to set it to just use the release directly, which we could include in our startup environment variables on head:

jeanconn-han8-v> export APP_ENV_ASCDS_STR='/home/ascds/DS.release/config/system/.ascrc'

The more correct fix would be to update the OS checking in the perl code and make new releases of all the perl stuff to match. But the env var might be the way we want to go.

javierggt commented 2 years ago

I don't get the git error on han8-v with a newly created environment with ska3-flight 2022.4.

taldcroft commented 2 years ago

I still get that error. Can you think of any other env vars to check for a difference?

ska3-han8-v$ cd ~/git/kadi
ska3-han8-v$ git pull
/bin/ssh: symbol lookup error: /bin/ssh: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

ska3-han8-v$ echo $LD_LIBRARY_PATH
/proj/sot/ska3/flight/lib:/proj/sot/ska3/flight/pgplot:/soft/SYBASE16.0/OCS-16_0/lib

ska3-han8-v$ which git
/bin/git

jeanconn commented 2 years ago

I would think this would be an issue where the remote (https vs ssh) gives different answers?

javierggt commented 2 years ago

I get the git error if I setup my environment using:

source /proj/sot/ska3/flight/bin/ska_envs.sh

and I do not get it if I set it up with

eval "$(/proj/sot/ska3/flight/bin/conda shell.bash hook)"

I'll continue digging.

javierggt commented 2 years ago

and the Ska.Shell error happens because the envs variable contains IFS:

In [4]: envs['IFS']
Out[4]: '$_mlIFS;'

This is not how it is set in my shell, but when one does get_env this is what I get.

jzuhone commented 2 years ago

@taldcroft I ran the acis_thermal_check tests on han8-v and they all passed. I'll try kadi next.

taldcroft commented 2 years ago

@jzuhone - I'm seeing acis_thermal_check tests passing now as well, though this time I ran with acis_thermal_check.test() instead of via our integration test suite. Maybe I'll re-try that.

jzuhone commented 2 years ago

@taldcroft that's how I ran them, with acis_thermal_check.test().

Separate question--how do I get the kadi tests to run without 401 errors from occweb?

taldcroft commented 2 years ago

The acis_thermal_check tests are also passing now via ska_testr, so I'll chalk up the previous failure as transient. 🤞

javierggt commented 2 years ago

Which ska3-flight version are you using for the acis_thermal_check tests? Because they failed in ska3-masters last night, and they failed in the ska3-matlab RC. So, what setup are you using?

If it is a problem of ska3-masters, I can check, but it was also ska3-matlab. At this time I would just like to know which setups pass.

javierggt commented 2 years ago

never mind, I was confused.

jzuhone commented 2 years ago

@taldcroft ok, now I can reproduce the kadi failure

taldcroft commented 2 years ago

@jzuhone - I'm working on the kadi failure. It looks really obscure, a floating point difference in the 11th decimal place of a quaternion in one sample of commands.

taldcroft commented 2 years ago

See https://github.com/sot/skare3/issues/857 for the kadi problem, which is unrelated to kadi but instead a floating point calculation miscompare in the last bit.

javierggt commented 2 years ago

I think we can close this already. Reopen otherwise.

sot / skare3