Problem: Hare is not tested by QA

[vvv]

Solution:

• prepare Hare rpm (#284)
• engage QA team

---

See also:

• EES Pacemaker HA HOWTO

[vvv]

closed

[vvv]

I've created #422 for the UUID issue.

[vvv]

mentioned in issue #422

[vvv]

@dmitriy.chumak wrote:

m0setup generates it

The funny thing is that Hare does not call m0setup...

$ git grep m0setup origin/master
$ 

[vvv]

changed the description

[andriytk]

The problem of non-unique FIDs was caused by the flaws in our software design and implementation. UUIDs design and implementation are not ours - feel the difference! ;)

As for the time spending on this discussion - agree. Let's stop on it.

[max-seagate]

The time spent on this discussion could be used to write a simple query to check uniqueness (a simple query in the middle of adding UUID). Look at this as at M0_ASSERT(), just to be 100% sure.

Let me remind you about the non-unique fids in A200. The reason was different, but the consequences were significant. I'm not telling that this may happen with node UUIDs, but still.

[andriytk]

@max-seagate.medved I mean mainly the sanity of this - is this really needed (to make sure that the generated UUIDs are really unique)? Isn't it overkill? Don't we have anything more important/urgent to do? Can't we return to this and easily implement it later if any collision will ever happen (of which I'm very unsure)?

cc @nikita.danilov

[max-seagate]

@andriy.tkachuk: Kidding? You will have hundreds of queries to the configuration database every second on large clusters (entrypoints, HA state changes etc.). I'm talking about a single query (with expected complexity logN) every time a node is added to the cluster. You're talking about a performance of something that is done once for every node in the cluster and for which no one have profiling information for. I think such kind of optimizations should be done when we have profiling information.

[andriytk]

@max-seagate.medved my question about Halon was in the context of providing the uniqueness for the UUID values, not that it just added it into the /etc/sysconfig/mero-kernel file.

This functionality (the uniqueness check) may seem simple for the small cluster configurations. But imagine the configurations of hundreds of thousands of the nodes. From the other side, the uuid generation was designed in such a way (as far as I understand) that the collisions probability should be ignorable. So having such a check seems like overkill to me.

[max-seagate]

Please read http://gitlab.mero.colo.seagate.com/mero/hare/issues/91#note_17379 and http://gitlab.mero.colo.seagate.com/mero/hare/issues/91#note_17399. Both of them have reference to the Halon code that does this.

I don't see any practical reason not to do a simple uniqueness check when a new node is added to the cluster. It's very simple, it doesn't require hundreds lines of the code and it will save a lot of headache if we ever have a collision. Is there any practical reason not to implement this check that I don't see?

[andriytk]

Did Halon do this? Or this is something we never did before but would be right to do?

How important/critical/practical it is given that the collisions "probability so small that it can normally be ignored" (from Wiki page above)?

[max-seagate]

how Hare should guarantee the uniqueness of these UUIDs

Hare manages the configuration database. It can check if the newly generated UUID exists in the database already - this way collisions can be completely avoided.

[andriytk]

Generate UUID for each node in the cluster during the initial bootstrap...

Sorry, I meant - how Hare should guarantee the uniqueness of these UUIDs and whether Halon did this.

See also https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions.

[max-seagate]

How Hare should do this? What algorithm do you suggest?

Generate UUID for each node in the cluster during the initial bootstrap and when the node is added to the cluster, save it in it's own database (Consul) and then write UUID to the configuration file each time mero-kernel is started.

Did Halon do this?

My initial comment in the discussion, start with "Halon does this with the following code:".

Again - what is Hare's responsibility here?

Hare's responsibility is to assign UUID to the nodes in the cluster and to show this UUID to whoever needs this. Right now the only consumer is mero-kernel. addbdump will need this when we start analyzing records from several nodes.

Is it supposed to provide this "well-knownness" somehow to someone? How, by what means?

Show it in the cluster status (and machine-readable representation of cluster status too), for example.

[andriytk]

Because node UUID should be unique in the cluster...

How Hare should do this? What algorithm do you suggest? Did Halon do this?

It's not just an UUID, it should be well-known node UUID that can be used by other tools...

Again - what is Hare's responsibility here? Is it supposed to provide this "well-knownness" somehow to someone? How, by what means?

I'm just trying to understand the whole picture here...

[max-seagate]

Because node UUID should be unique in the cluster and it's a configuration parameter. It's not just an UUID, it should be well-known node UUID that can be used by other tools to identify the node (example: addb). Hare manages Mero configuration, so it's a responsibility of Hare to configure Mero properly.

[andriytk]

I'm struggling to understand - why Hare should do this? Why can't Mero do this by itself on, say, rpm install?

[chumakd]

@andriy.tkachuk m0setup generates it and writes to /etc/mero/genders file, which is a rudiment of the early days of ActiveArcive, when we used to run Mero w/o Halon :wink:

[andriytk]

Hmm, I wonder how and why does it work without this file:

[vagrant@pod-c1 ~]$ lsmod | grep m0
m0ctl                  48916  0 
m0mero              13856252  1 m0ctl
m0gf                  121518  1 m0mero
lnet                  484170  3 m0mero,ksocklnd
[vagrant@pod-c1 ~]$ ls -l /etc/sysconfig/mero-kernel
ls: cannot access /etc/sysconfig/mero-kernel: No such file or directory

@dmitriy.chumak any idea?

[max-seagate]

dmesg from the crash: vmcore-dmesg.txt.

Root cause of the crash:

[ 5165.112926] mero[13453]:  fa28  ERROR  [/root/rpmbuild/BUILD/mero-1.4.0/lib/uuid.c:65:parse_hex]  <! rc=-22
[ 5165.112929] mero[13453]:  fa68  ERROR  [/root/rpmbuild/BUILD/mero-1.4.0/lib/uuid.c:90:m0_uuid_parse]  <! rc=-22
[ 5165.112931] mero[13453]:  fac0  ERROR  [/root/rpmbuild/BUILD/mero-1.4.0/lib/uuid.c:136:m0_node_uuid_init]  Unable to parse node UUID string
[ 5165.112932] subsystem libm0 init failed: rc = -22
[ 5165.112960] BUG: unable to handle kernel NULL pointer dereference at           (null)
[ 5165.112991] IP: [<ffffffffc0ce08b3>] m0_addb2_global_thread_leave+0x13/0x140 [m0mero]
[ 5165.113142] PGD 8000002fd3fb9067 PUD 2fcf3d3067 PMD 0

mero-kernel requires UUID string in MERO_NODE_UUID parameter:

 176         local uuid=${MERO_NODE_UUID:-$(m0_genders_value_of m0_uuid)}
...
 192         insmod $(m0_path_to m0mero.ko) $m0mero_params || {
 193             rmmod $(m0_path_to m0gf.ko)
 194             m0_exit "Failed to load m0mero module"
 195         }

Halon does this with the following code:

> gg MERO_NODE_UUID
mero-halon/src/lib/HA/Services/Mero.hs:436:        [ ("MERO_NODE_UUID", UUID.toString $ mkcNodeUUID (mcKernelConfig conf))
mero-halon/tests/HA/Services/Mero/Mock.hs:547:        [ ("MERO_NODE_UUID", UUID.toString $ mkcNodeUUID (mcKernelConfig conf)) ]

433     -- Kernel
434     startKernel = liftIO $ do
435       _ <- SystemD.sysctlFile "mero-kernel"
436         [ ("MERO_NODE_UUID", UUID.toString $ mkcNodeUUID (mcKernelConfig conf))
437         ]
438       SystemD.startService "mero-kernel"

Hare should also do this.

The kernel panic bug is a Mero bug and it should be fixed separately.

[Kalpesh-Chhajed]

Thanks @vvv . This help me to fix the lnet issue.

Once lnet is up and running, when i try to bootstrap the cluster every time , i am facing kernel crash.

I tried manually starting the mero-kernel and its resulting crash too.

I do not know how to analyze the crash so need some help to debug the crash

[vvv]

@kalpesh.chhajed Run this shell code:

# `lnetctl import /etc/lnet.conf` will fail unless /etc/lnet.conf
# exists and is a valid YAML document.  YAML document cannot be empty;
# document separator ('---') is the shortest valid YAML document.
if [[ ! -s /etc/lnet.conf ]]; then
    sudo tee /etc/lnet.conf <<<'---' >/dev/null
fi

Kudos to @dmitriy.chumak for the solution! (BTW, it's incorporated into m0vg.)

[Kalpesh-Chhajed]

LNet seems to be not working.

[root@eosnode-2 ~]# lsmod | grep lnet
lnet                  484170  1
libcfs                415749  1 lnet
[root@eosnode-2 ~]# lctl network up
LNET configured
[root@eosnode-2 ~]# lctl list_nids
[root@eosnode-2 ~]#

lctl list_nids shows nothing, which doesn't seem right.

[Kalpesh-Chhajed]

I followed the procedure described in the README.md.

hctl bootstrap --mkfs ~/CDF.yaml fails:

[root@eosnode-2 ~]# hctl bootstrap --mkfs ~/CDF.yaml
2019-10-29 19:56:22: Generating cluster configuration... Ok.
2019-10-29 19:59:04: Starting Consul server agent on this node........................................... Ok.
2019-10-29 19:59:48: Importing configuration into the KV Store... Ok.
2019-10-29 19:59:48: Starting Consul agents on remaining cluster nodes... Ok.
2019-10-29 19:59:48: Update Consul agents configs from the KV Store... Ok.
2019-10-29 19:59:49: Waiting for the RC Leader to be elected...... Ok.
2019-10-29 19:59:53: Starting Mero (phase1)... A dependency job for hare-hax.service failed. See 'journalctl -xe' for details.

journalctl output:

-- Unit system-mero\x2dtrace.slice has begun starting up.
Oct 29 19:59:53 eosnode-2 kernel: libcfs: loading out-of-tree module taints kernel.
Oct 29 19:59:53 eosnode-2 kernel: libcfs: module verification failed: signature and/or required key missing - tainting kernel
Oct 29 19:59:53 eosnode-2 kernel: LNet: HW NUMA nodes: 2, HW CPU cores: 48, npartitions: 2
Oct 29 19:59:53 eosnode-2 kernel: alg: No test for adler32 (adler32-zlib)
Oct 29 19:59:53 eosnode-2 hare-consul[58002]: 2019/10/29 19:59:53 [WARN] agent: Check "service:6" is now warning
Oct 29 19:59:53 eosnode-2 hare-consul[58002]: 2019/10/29 19:59:53 [WARN] agent: Check "service:12" is now warning
Oct 29 19:59:53 eosnode-2 hare-consul[58002]: 2019/10/29 19:59:53 [WARN] agent: Check "service:9" is now warning
Oct 29 19:59:54 eosnode-2 lnetctl[60252]: yaml:
Oct 29 19:59:54 eosnode-2 lnetctl[60252]: - builder:
Oct 29 19:59:54 eosnode-2 lnetctl[60252]: errno: -1
Oct 29 19:59:54 eosnode-2 systemd[1]: lnet.service: main process exited, code=exited, status=255/n/a
Oct 29 19:59:54 eosnode-2 lnetctl[60252]: descr: Failed to handle token:2 [state=1, rc=-1]
Oct 29 19:59:54 eosnode-2 systemd[1]: Failed to start lnet management.
-- Subject: Unit lnet.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit lnet.service has failed.
--
-- The result is failed.
Oct 29 19:59:54 eosnode-2 systemd[1]: Dependency failed for Mero kernel module.
-- Subject: Unit mero-kernel.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mero-kernel.service has failed.
--
-- The result is dependency.
Oct 29 19:59:54 eosnode-2 systemd[1]: Dependency failed for HAX daemon for Hare.
-- Subject: Unit hare-hax.service has failed 

cc @vvv

[vvv]

assigned to @vvv

[andriytk]

changed milestone to %3

[vvv]

unassigned @vvv

[andriytk]

@dmitriy.chumak clarified - we do need rpms in order to give Hare to QAs. Two main reasons:

• that they don't build Mero (and we depend on Mero source code atm);
• they don't work with sources from gitlab.

In other words - they take only already built binaries for testing. Well, I did not know this.

[andriytk]

As for the 2nd part (engaging QA team) - I'm assigning this to you @vvv as you are geographically close to them these days and it seems like a good opportunity to show them a master class. :)

[andriytk]

assigned to @vvv

[andriytk]

I'm not sure this issue depends on rpm. QAs can take Hare directly from gitlab and use the README as a guideline, can't they?

[vvv]

changed the description