'Referenced RSE not reachable' error

[MattSlm]

facing this error, already stuck for hours right now! Was curious if anybody has any idea about the cause or have faced it before.

None of these returns an error

docker stop $(sudo docker ps -a -q)
docker rm $(sudo docker ps -a -q)
docker-compose down --volumes
docker-compose -f docker-compose-with-rucio-monit.yml up -d
docker-compose -f docker-compose-with-rucio-monit.yml exec mininet python3 /utils/rucio_example.py

However rucio's test on containernet fails to replicate generating these lines:

Details: [ERROR] Operation expired
2022-04-13 20:28:21,093 INFO    Preparing upload for file file2
2022-04-13 20:28:21,175 INFO    Successfully added replica in Rucio catalogue at XRD1
2022-04-13 20:28:21,264 INFO    Successfully added replication rule at XRD1
2022-04-13 20:28:36,349 ERROR   Referenced RSE not reachable.
Details: Referenced RSE not reachable.
Details: [ERROR] Operation expired
2022-04-13 20:28:38,550 INFO    Preparing upload for file file3
2022-04-13 20:28:38,599 INFO    Successfully added replica in Rucio catalogue at XRD2
2022-04-13 20:28:38,638 INFO    Successfully added replication rule at XRD2
2022-04-13 20:28:53,731 ERROR   Referenced RSE not reachable.
Details: Referenced RSE not reachable.
Details: [ERROR] Operation expired
2022-04-13 20:28:55,865 INFO    Preparing upload for file file4
2022-04-13 20:28:55,911 INFO    Successfully added replica in Rucio catalogue at XRD2
2022-04-13 20:28:55,947 INFO    Successfully added replication rule at XRD2
2022-04-13 20:29:11,020 ERROR   Referenced RSE not reachable.
Details: Referenced RSE not reachable.
Details: [ERROR] Operation expired

@fno2010 really appreciate it if you can take a look.

Pings are successful among containers

[MattSlm]

Apparently it is due to RSEAccessDenied.

[fno2010]

@MattSlm could you post the full log? It looks like the xrootd service was not configured properly. I want to see if the xrootd info was shown correctly.

Also, I see the rucio docker containers (rucio/rucio-dev, rucio/xrootd) are updated recently. Not sure if they made any critical changes that may affect the xrootd setup. If you have pulled the latest rucio docker containers, could you switch the rucio repo to the master branch and try again?

[MattSlm]

I have resolved my issue, here are some fixes which might also resolve same issues: 1- If you go through the env setup document, after completing setup 1- network volumes interfere with the networks for the second experiment. so

docker-compose down -v

will resolve this issue. 2- In the makefile, two lines are commented out which makes the makes ineffective. In my case the above error got resolved with un-commenting those. 3- cd docker inside env setup document is put after git clone commands, this makes empty directories to be mounted inside ssh machine, in that case ssh container fails to start. 4- In some docker-compose versions explicit versioning is required to avoid no option found for module errors. this can be fixed with adding

version: '3.3' 

at the begging of all docker files.

[giralt]

Thank you @MattSlm , do you plan to issue a pull request with the suggested fixes so we can review and merge them into main branch? thks.

[MattSlm]

Update: Another issue still exist that I forgot to mention yesterday. A minor fix on s-flow image is needed. It seems that the mininet-sadhboard is not installed on the image. so

docker-compose -f docker-compose-with-rucio-monit.yml exec sflow /sflow-rt/get-app.sh sflow-rt mininet-dashboard

is not working. I am fixing this.

[giralt]

Thank you @MattSlm , could you then reopen this ticket? Then once you have a fix for it, please issue a pull request. The ticket should not be closed until the PR is completed and the code committed. Thanks!

[fno2010]

Update: Another issue still exist that I forgot to mention yesterday. A minor fix on s-flow image is needed. It seems that the mininet-sadhboard is not installed on the image. so

docker-compose -f docker-compose-with-rucio-monit.yml exec sflow /sflow-rt/get-app.sh sflow-rt mininet-dashboard

is not working. I am fixing this.

@MattSlm I don't understand this issue. The mininet-dashbard doesn't have to be installed on the image. Because the command that you typed is just to install mininet-dashboard. It will clone the sflow-rt/mininet-dashboard repo into the /sflow-rt/app directory of the container. What error you saw when you typed this command?

[MattSlm]

@fno2010 this is the error: error: app/mininet-dashboard-master/LICENSE (No such file or directory)