search

ska-telescope / tmc-prototype

The repository for TMC evolutionary prototype.
BSD 3-Clause "New" or "Revised" License
2 stars 1 forks source link

make up fails #12

Closed Matteo04052017 closed 5 years ago

Matteo04052017 commented 5 years ago

I have just downloaded the project and called the make up to check whether the devices are working. Basically the only device working is the dishmaster. This is the result of the command "docker ps -a": CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e80895592b55 nexus.engageska-portugal.pt/tango-example/tmcprototype:latest "sh -c 'wait-for-it.…" 46 seconds ago Exited (255) 35 seconds ago tmc-prototype_centralnode_1 ba9b5eb64e49 nexus.engageska-portugal.pt/tango-example/tmcprototype:latest "sh -c 'wait-for-it.…" 47 seconds ago Exited (255) 35 seconds ago tmc-prototype_subarraynode_1 dd699a0db056 nexus.engageska-portugal.pt/tango-example/tmcprototype:latest "sh -c 'wait-for-it.…" 47 seconds ago Exited (255) 46 seconds ago tmc-prototype_dishleafnode_1 a66d4b60dc15 nexus.engageska-portugal.pt/tango-example/tmcprototype:latest "sh -c 'wait-for-it.…" 48 seconds ago Up 47 seconds tmc-prototype_dishmaster_1 12773cf4c148 nexus.engageska-portugal.pt/ska-docker/tango-cpp:latest "/usr/local/bin/wait…" 49 seconds ago Up 48 seconds tmc-prototype_databaseds_1 fcaf8bd099dd nexus.engageska-portugal.pt/ska-docker/tango-db:latest "docker-entrypoint.s…" 50 seconds ago Up 48 seconds 3306/tcp tmc-prototype_tangodb_1 38ab146cd438 jumanjiman/rsyslog "rsyslogd -n -f /etc…" 2 minutes ago Up 2 minutes 514/tcp, 514/udp rsyslog-tmcprototype 682623edb58a 3168bb55dd62 "/venv/bin/python /a…" 13 minutes ago Created tmcprototype-test-cache bed5a968d264 nexus.engageska-portugal.pt/ska-docker/tango-pogo:latest "/usr/local/bin/pogo" 2 hours ago Created pogo ffdb7055dcf4 nexus.engageska-portugal.pt/ska-docker/tango-java:latest "/usr/local/bin/wait…" 14 hours ago Created jive bf755c0c3a6f nexus.engageska-portugal.pt/ska-docker/tango-java:latest "/usr/local/bin/wait…" 14 hours ago Exited (0) 14 hours ago tangotest 8dd3c8038e5f nexus.engageska-portugal.pt/ska-docker/tango-java:latest "/usr/local/bin/wait…" 14 hours ago Created astor 3f7ae4fa29c2 nexus.engageska-portugal.pt/ska-docker/tango-rest:latest "/usr/local/bin/wait…" 14 hours ago Created tango-rest 16ed417b64ec nexus.engageska-portugal.pt/ska-docker/tango-java:latest "/usr/local/bin/wait…" 14 hours ago Created logviewer 6e050d40cbc0 nexus.engageska-portugal.pt/ska-docker/tango-itango:latest "/usr/local/bin/wait…" 14 hours ago Created itango d1bff4c363ac nexus.engageska-portugal.pt/ska-docker/tango-starter:latest "/usr/local/bin/wait…" 14 hours ago Created starter-example f27fdae4341e nexus.engageska-portugal.pt/ska-docker/tango-cpp:latest "/usr/local/bin/wait…" 14 hours ago Up 2 hours databaseds 1f6bf179f223 nexus.engageska-portugal.pt/ska-docker/tango-db:latest "docker-entrypoint.s…" 14 hours ago Up 2 hours tangodb

For instance the dishleagnode exit after 46 seconds. If I check the logs, it says: wait-for-it.sh: waiting 30 seconds for databaseds:10000 wait-for-it.sh: databaseds:10000 is available after 0 seconds

This means that the device is able to see the database but it exit anyway. I suppose the problem is in the command "tango_admin --add-property test/DishLeafNode/01 DishMasterFQDN mid_d0001/elt/master" which make the device fails.

Did you think of adding a configuration script instead of using the tango-admin?

jyotinpo commented 5 years ago

Thanks for reporting this issue. I need to debug this in details. However, it is running fine in the CI pipeline without issues. We will be adding configuration script but will use tango-admin for now.

Matteo04052017 commented 5 years ago

well, the fact that the CI pipeline is running fine without issues is not really true. For the dishleafnode, for instance, I see some failures even if the pipeline says ok (https://gitlab.com/ska-telescope/tmc-prototype/-/jobs/182389422). If you just call the make up you will notice that the add-property is not working and that you're missing a library (katpoint). The fact that the pipeline work with no issue is due to two missing lines to the top of the ‘test-harness/Makefile’:

SHELL = /bin/bash .SHELLFLAGS = -o pipefail -c

There is a pull request by Stewart for the lmc-base-classes that I think should go also in this repo. I suggest you to add them to the Makefile to have some reality in the pipeline.

Matteo04052017 commented 5 years ago

As alternative to the tango_admin, I made (for my test with your code) a simple script which configure all the DB. If you lunch it, before the first device starts, it will make you remove the tango_admin command. 

import tango

# add device servers
db = Database()

dev_info = DbDevInfo()
dev_info.name = 'mid_d0001/elt/master'
dev_info._class = 'DishMaster'
dev_info.server = 'DishMaster/test'
db.add_device(dev_info)

dev_info = DbDevInfo()
dev_info.name = 'ska_mid/tm_leaf_node/d0001'
dev_info._class = 'DishLeafNode'
dev_info.server = 'DishLeafNode/test'
db.add_device(dev_info)

dev_info = DbDevInfo()
dev_info.name = 'ska_mid/tm_subarray_node/1'
dev_info._class = 'SubarrayNode'
dev_info.server = 'SubarrayNode/test'
db.add_device(dev_info)

dev_info = DbDevInfo()
dev_info.name = 'ska_mid/tm_subarray_node/2'
dev_info._class = 'SubarrayNode'
dev_info.server = 'SubarrayNode/test'
db.add_device(dev_info)

dev_info = DbDevInfo()
dev_info.name = 'ska_mid/tm_central/central_node'
dev_info._class = 'CentralNode'
dev_info.server = 'CentralNode/test'
db.add_device(dev_info)

# add device properties
dev = tango.DeviceProxy("test/DishLeafNode/01")
if (not dev.get_property("DishMasterFQDN")['DishMasterFQDN']):
    dev.put_property({"DishMasterFQDN":"mid_d0001/elt/master"})

dev = tango.DeviceProxy("ska_mid/tm_subarray_node/1")
if (not dev.get_property("DishLeafNodePrefix")['DishLeafNodePrefix']):
    dev.put_property({"DishLeafNodePrefix":"ska_mid/tm_leaf_node/d"})

dev = tango.DeviceProxy("ska_mid/tm_subarray_node/2")
if (not dev.get_property("DishLeafNodePrefix")['DishLeafNodePrefix']):
    dev.put_property({"DishLeafNodePrefix":"ska_mid/tm_leaf_node/d"})
jyotinpo commented 5 years ago

Hi @Matteo04052017,

I am working on this bug on priority. I will keep posting updates on this issue as comments here. You were right about the katpoint dependency. Actually, I had added in the setup.py file of DishLeafNode package but not added to the Pipfile. Hence, it is not installed when the docker image is created but installed only when the test cases of DishLeafNode is executed.

Locally, I make some changes to install katpoint package when the Docker image is created. And now the output of docker ps -a returns following:

CONTAINER ID        IMAGE                                                           COMMAND                  CREATED             STATUS                       PORTS               NAMES
ab0f4ea508b0        nexus.engageska-portugal.pt/tango-example/tmcprototype:latest   "sh -c 'wait-for-it.…"   3 minutes ago       Exited (255) 3 minutes ago                       tmc-prototype_centralnode_1
14967aa170ce        nexus.engageska-portugal.pt/tango-example/tmcprototype:latest   "sh -c 'wait-for-it.…"   3 minutes ago       Up 3 minutes                                     tmc-prototype_subarraynode_1
d2fa0edb5940        nexus.engageska-portugal.pt/tango-example/tmcprototype:latest   "sh -c 'wait-for-it.…"   3 minutes ago       Up 3 minutes                                     tmc-prototype_dishleafnode_1
1d5bf7a72cba        nexus.engageska-portugal.pt/tango-example/tmcprototype:latest   "sh -c 'wait-for-it.…"   3 minutes ago       Up 3 minutes                                     tmc-prototype_dishmaster_1
2952727b0f0b        nexus.engageska-portugal.pt/ska-docker/tango-cpp:latest         "/usr/local/bin/wait…"   6 days ago          Up 19 minutes                                    tmc-prototype_databaseds_1
1087596463f3        nexus.engageska-portugal.pt/ska-docker/tango-db:latest          "docker-entrypoint.s…"   6 days ago          Up 19 minutes                3306/tcp            tmc-prototype_tangodb_1
e7156c62d32b        jumanjiman/rsyslog                                              "rsyslogd -n -f /etc…"   6 days ago          Up 19 minutes                514/tcp, 514/udp    rsyslog-tmcprototype
967e9d6376c1        0f520305ae11                                                    "/bin/sh -c 'buildDe…"   2 weeks ago         Exited (100) 2 weeks ago                         agitated_keldysh
7451fd87f392        0f520305ae11                                                    "/bin/sh -c 'buildDe…"   2 weeks ago         Exited (100) 2 weeks ago                         hungry_ride
ef4b506c23b0        0f520305ae11                                                    "/bin/sh -c 'buildDe…"   2 weeks ago         Exited (2) 2 weeks ago                           dreamy_hofstadter
59c1fae9fa99        0f520305ae11                                                    "/bin/sh -c 'buildDe…"   2 weeks ago         Exited (100) 2 weeks ago                         lucid_yonath
79223ef7a91c        41ab524c1fe2                                                    "/bin/sh -c 'pipenv …"   2 weeks ago         Exited (1) 2 weeks ago                           quirky_shockley
b25ee6e88a29        9ea27399331e                                                    "/bin/sh -c 'buildDe…"   2 weeks ago         Exited (1) 2 weeks ago                           mystifying_dhawan
e46d20c0b5fc        9ea27399331e                                                    "/bin/sh -c 'buildDe…"   2 weeks ago         Exited (1) 2 weeks ago                           elegant_bose
dc427581ed11        9ea27399331e                                                    "/bin/sh -c 'buildDe…"   2 weeks ago         Exited (1) 2 weeks ago                           inspiring_haibt
785d26bcdc61        9ea27399331e                                                    "/bin/sh -c 'buildDe…"   2 weeks ago         Exited (1) 2 weeks ago                           vigorous_ellis
7f0605578733        9ea27399331e                                                    "/bin/sh -c 'buildDe…"   2 weeks ago         Exited (100) 2 weeks ago                         zealous_dewdney
e243e9418882        9ea27399331e                                                    "/bin/sh -c 'buildDe…"   2 weeks ago         Exited (1) 2 weeks ago                           recursing_robinson
79e76cb592fb        9ea27399331e                                                    "/bin/sh -c 'buildDe…"   2 weeks ago         Exited (100) 2 weeks ago                         amazing_saha
74d7ecc73255        ska-registry.av.it.pt/tango-example/lmcbaseclasses:latest       "/venv/bin/python /a…"   5 weeks ago         Created                                          lmcbaseclasses-test-cache
ffb5facbac99        9b314edf30f9                                                    "/venv/bin/python /a…"   6 weeks ago         Created                                          determined_cohen
e627611d3151        82db9abdf453                                                    "/venv/bin/python /a…"   6 weeks ago         Created                                          tmcprototype-test-cache
e7e8b705f9df        ecf4779095b8                                                    "sh -c 'wait-for-it.…"   6 weeks ago         Exited (255) 6 weeks ago                         dishmaster_dishmaster_1
b9551435780e        ska-registry.av.it.pt/ska-docker/tango-cpp:latest               "/usr/local/bin/wait…"   6 weeks ago         Exited (255) 6 weeks ago                         dishmaster_databaseds_1
15d1f3552195        ska-registry.av.it.pt/ska-docker/tango-db:latest                "docker-entrypoint.s…"   6 weeks ago         Exited (255) 6 weeks ago     3306/tcp            dishmaster_tangodb_1
3bed8f6f7e4b        jumanjiman/rsyslog                                              "rsyslogd -n -f /etc…"   6 weeks ago         Exited (255) 6 weeks ago     514/tcp, 514/udp    rsyslog-dishmaster
819ad73b2ae1        hello-world                                                     "/hello"                 6 weeks ago         Exited (0) 6 weeks ago                           distracted_shirley
3882a3d4d9bd        hello-world                                                     "/hello"                 6 weeks ago         Exited (0) 6 weeks ago                           compassionate_johnson
22baba3557f3        hello-world                                                     "/hello"                 6 weeks ago         Exited (0) 6 weeks ago                           unruffled_burnell
7c03ba84c91c        hello-world                                                     "/hello"                 6 weeks ago         Exited (0) 6 weeks ago                           stupefied_goldwasser

There is an introduction of json package at CentralNode which has to be added due to which the CentralNode fails. I will add it now. I will also check the device property issue of DishLeafNode which you have reported.

May I know how do you check failure logs as you specified earlier? Is it the same terminal log when ran in CI pipeline?

Matteo04052017 commented 5 years ago

I put two "echo" commands: one before the add-property and one after it. Eventually I completely deleted the use of the tango-admin from the integration environment in favor of a python configuration script. I have also checked the github of the tango-admin and, actually, it should work. It is something wired... Let me know if you have any news on this topic but I don't like allot that a component configure itself...

jyotinpo commented 5 years ago

Hi @Matteo04052017,

Can you please confirm if this issue is resolved? The configuration script is not included yet in this merge request. We will add a user story for it in the next sprint and commit.

Matteo04052017 commented 5 years ago

Are you sure that, without the configuration script, it works? in other words, is the "put-property" of the tango-admin working? I would prefer to have make up and configuration together if possible...

jyotinpo commented 5 years ago

Yes, the way of putting property using tango-admin is working. make up is running fine without issues. I will surely update the configuration script and include it later. It is okay if we keep this issue open until the configuration script is not included. CI Pipeline is executed successfully. For now, I am just looking forward to your confirmation whether make up of the current tmc-prototype master is working fine at your end or not?

Matteo04052017 commented 5 years ago

I'll give it a try and I'll let you know..

Matteo04052017 commented 5 years ago

I have execute the make up command on the master branch with no success. This is the result of a docker ps command: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 9160ae4e6744 nexus.engageska-portugal.pt/tango-example/tmcprototype:latest "sh -c 'wait-for-it.…" About a minute ago Up About a minute tmc-prototype_dishmaster_1 81f14b174e46 nexus.engageska-portugal.pt/ska-docker/tango-cpp:latest "/usr/local/bin/wait…" About a minute ago Up About a minute tmc-prototype_databaseds_1 9244c6f6e7e4 nexus.engageska-portugal.pt/ska-docker/tango-db:latest "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp tmc-prototype_tangodb_1 92e11c860a43 jumanjiman/rsyslog "rsyslogd -n -f /etc…" About a minute ago Up About a minute 514/tcp, 514/udp rsyslog-tmcprototype

I can see working the following containers: rsyslog-tmcprototype, tmc-prototype_tangodb_1, tmc-prototype_databaseds_1 and tmc-prototype_dishmaster_1.

I have also taken a look to the pipeline and this job is really bad: https://gitlab.com/ska-telescope/tmc-prototype/-/jobs/193365418 Why is it successfully?

For me, this pipeline is not working and this issue does not seem to be resolved.

jyotinpo commented 5 years ago

It is strange. I had created a new pipeline for the master branch and tried executing it. It is running successfully. I did encounter failure for the pages job, but when I tried re-try, it ran successfully. I think there is some issue in the infrastructure. I am in communication with Dzianis Bartashevich and seems that he fixed something.

Here's the link to the pipeline which ran successfully: https://gitlab.com/ska-telescope/tmc-prototype/pipelines/55940301

Can you try one more time with a fresh setup, please?

Matteo04052017 commented 5 years ago

Yes, something happened to the infrastructure. Apparently there were too many docker networks called "tmc-prototype-something". Perhaps those were the tests for the makefile bug fixing. Anyway, now the make up is working: let's close the ticket. .