search

metacpan / metacpan-docker

DEVELOPMENT Docker configs for MetaCPAN
Artistic License 2.0
26 stars 23 forks source link

Initial Setup fails with: "Duplicate mount point: /CPAN" #87

Closed domibay-hugo closed 3 years ago

domibay-hugo commented 3 years ago

I'm following along the guide at CPAN Docker Quick Start to do the initial setupl of my development environment. After git-clone the next suggested command docker-compose exec api index-cpan.sh fails with:

$ docker-compose exec api index-cpan.sh
ERROR: No container found for api_1

So I concluded the environment was not initialized correctly. I started to look in the docker-compose.yml figure out which Images needed to be build and started to build the Images for the API as found in:

  api:
    depends_on:
      - elasticsearch
      - pgdb
      - traefik

So I built the Images:

Still the Error persisted:

$ docker-compose exec api index-cpan.sh
ERROR: No container found for api_1
$ docker-compose exec api_test prove -lvr \
>   t/00_setup.t \
>   t/01_darkpan.t \
>   t/api/controller/cover.t
ERROR: No container found for api_test_1
$ docker-compose exec api /bin/bash
ERROR: No container found for api_1

So I tried to bring up the API with docker-compose up api This concludes in Error, too:

$ docker-compose up api
metacpan_elasticsearch_1 is up-to-date
metacpan_pgdb_1 is up-to-date
metacpan_traefik_1 is up-to-date
Recreating metacpan_api_1 ... error

ERROR: for metacpan_api_1  Cannot create container for service api: Duplicate mount point: /CPAN

ERROR: for api  Cannot create container for service api: Duplicate mount point: /CPAN
ERROR: Encountered errors while bringing up the project.

In the docker-compose.yml I see that the api Service has a Volume /CPAN:

  api:
# [...]
    volumes:
      - type: volume
        source: cpan
        target: /CPAN

But I don't know what is its purpose and why should it be duplicate.

domibay-hugo commented 3 years ago

From the logic in docker-compose.yml I assumed that cpan is a local NEW directory that I need to create:

  api:
# [...]
    volumes:
      - type: volume
        source: cpan
        target: /CPAN
      - type: bind
        source: ./src/metacpan-api
        target: /metacpan-api

So I created the directory and I was able to start up the cluster.

# docker-compose up -d api
Creating network "metacpan_traefik-network" with the default driver
Creating network "metacpan_default" with the default driver
Creating network "metacpan_web-network" with the default driver
Creating network "metacpan_elasticsearch" with the default driver
Creating network "metacpan_database" with the default driver
Creating metacpan_traefik_1       ... done
Creating metacpan_api_1 ... done
Creating metacpan_pgdb_1          ... done
Creating metacpan_elasticsearch_1 ... 
Creating metacpan_api_1 ...

From ./bin/index-cpan.sh I learned that /CPAN is the Download Directory for the Module Source Code of the CPAN Modules:

./bin/run bin/metacpan release /CPAN/authors/id/

Still the Command docker-compose exec api /bin/bash fails:

# docker-compose exec api /bin/bash
ERROR: No container found for api_1

From this source I learned that the API Cluster must be running for this command to succeed docker-compose - ERROR: No container found

So after I started up the API Cluster with docker-compose up -d api I was able to run the Index Comand docker-compose exec api index-cpan.sh

# docker-compose up -d api
Creating network "metacpan_traefik-network" with the default driver
Creating network "metacpan_default" with the default driver
Creating network "metacpan_web-network" with the default driver
Creating network "metacpan_elasticsearch" with the default driver
Creating network "metacpan_database" with the default driver
Creating metacpan_traefik_1       ... done
Creating metacpan_api_1 ... done
Creating metacpan_pgdb_1          ... done
Creating metacpan_elasticsearch_1 ... 
Creating metacpan_api_1 ... 
# docker-compose exec api /bin/bash
root@c0d03d597b8b:/metacpan-api# hostname
c0d03d597b8b
root@c0d03d597b8b:/metacpan-api# ls -lah /CPAN
total 0
drwxr-xr-x. 2 root root  6 Jul 12 07:44 .
drwxr-xr-x. 1 root root 27 Jul 12 10:14 ..

Unfortunately the Command index-cpan.sh fails because it uses rsync to download the files and this needs a SSH Access to the perl.org servers which New Contributors don't have:

# docker-compose exec api index-cpan.sh
*** Warning ***: this will delete EVERYTHING and re-create the (empty) indexes
Are you sure you want to do this (type "YES" to confirm) ? yes
bye.
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
2021/07/12 10:16:19 E release: Dunno what /CPAN/authors/id/ is
2021/07/12 10:16:20 W release: /CPAN/modules/06perms.txt could not be found.
[Missing] ** [http://elasticsearch:9200]-[404] [index_not_found_exception] no such index, with: {"index":"cpan","resource.id":"cpan","resource.type":"index_or_alias"}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/lib/perl5/site_perl/5.30.1/ElasticSearchX/Model/Index.pm line 120. With vars: {'body' => {'status' => 404,'error' => {'index' => 'cpan','resource.id' => 'cpan','root_cause' => [{'resource.id' => 'cpan','index' => 'cpan','reason' => 'no such index','resource.type' => 'index_or_alias','type' => 'index_not_found_exception'}],'type' => 'index_not_found_exception','resource.type' => 'index_or_alias','reason' => 'no such index'}},'request' => {'method' => 'POST','ignore' => [],'body' => undef,'qs' => {},'serialize' => 'std','path' => '/cpan/_refresh'},'status_code' => 404}
Can't open /CPAN/modules/02packages.details.txt.gz: cannot open file '/CPAN/modules/02packages.details.txt.gz': No such file or directory at /usr/local/lib/perl5/site_perl/5.30.1/Parse/CPAN/Packages/Fast.pm line 97.
File does not exist: /CPAN/authors/00whois.xml at /metacpan-api/bin/../lib/MetaCPAN/Script/Author.pm line 54.
the file you specified with 'path' doesn't exist at /metacpan-api/bin/../lib/MetaCPAN/Script/Permission.pm line 35.

So I don't know where I could get these files from to actually get my Development Enviroment running.

oalders commented 3 years ago

Once this has been wrapped up, maybe we could patch the docs to make it a bit clearer that you need to have the containers running before docker-compose exec will be useful. I have run into this myself.

As far as rsync goes, you should be able to sync from this machine without any specific permissions. Can you try adding more vs to the relevant line inbin/partial-cpan-mirror.sh to see what the exact issue is? Something like RSYNC='/usr/bin/rsync -avvv --delete --relative'

domibay-hugo commented 3 years ago

Yes, I did the change in bin/partial-cpan-mirror.sh and rebuild the container with docker-compose build api.

Now when I run the command it shows:

# docker-compose exec api index-cpan.sh
*** Warning ***: this will delete EVERYTHING and re-create the (empty) indexes
Are you sure you want to do this (type "YES" to confirm) ? yes
bye.
rsync cmd: '/usr/bin/rsync -avvv --delete --relative'
opening tcp connection to cpan-rsync.perl.org port 873
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
[Receiver] _exit_cleanup(code=10, file=clientserver.c, line=127): about to call exit(10)
opening tcp connection to cpan-rsync.perl.org port 873
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
[Receiver] _exit_cleanup(code=10, file=clientserver.c, line=127): about to call exit(10)
opening tcp connection to cpan-rsync.perl.org port 873
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
[Receiver] _exit_cleanup(code=10, file=clientserver.c, line=127): about to call exit(10)
opening tcp connection to cpan-rsync.perl.org port 873
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
[Receiver] _exit_cleanup(code=10, file=clientserver.c, line=127): about to call exit(10)
opening tcp connection to cpan-rsync.perl.org port 873
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
[Receiver] _exit_cleanup(code=10, file=clientserver.c, line=127): about to call exit(10)
opening tcp connection to cpan-rsync.perl.org port 873
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
[Receiver] _exit_cleanup(code=10, file=clientserver.c, line=127): about to call exit(10)
opening tcp connection to cpan-rsync.perl.org port 873
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
[Receiver] _exit_cleanup(code=10, file=clientserver.c, line=127): about to call exit(10)
opening tcp connection to cpan-rsync.perl.org port 873
rsync: failed to connect to cpan-rsync.perl.org (45.54.12.31): Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(127) [Receiver=3.1.3]
[Receiver] _exit_cleanup(code=10, file=clientserver.c, line=127): about to call exit(10)
2021/07/12 14:36:26 E release: Dunno what /CPAN/authors/id/ is
2021/07/12 14:36:27 W release: /CPAN/modules/06perms.txt could not be found.
[Missing] ** [http://elasticsearch:9200]-[404] [index_not_found_exception] no such index, with: {"index":"cpan","resource.id":"cpan","resource.type":"index_or_alias"}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/local/lib/perl5/site_perl/5.30.1/ElasticSearchX/Model/Index.pm line 120. With vars: {'status_code' => 404,'request' => {'body' => undef,'serialize' => 'std','method' => 'POST','qs' => {},'path' => '/cpan/_refresh','ignore' => []},'body' => {'error' => {'reason' => 'no such index','type' => 'index_not_found_exception','resource.type' => 'index_or_alias','resource.id' => 'cpan','index' => 'cpan','root_cause' => [{'reason' => 'no such index','resource.type' => 'index_or_alias','type' => 'index_not_found_exception','resource.id' => 'cpan','index' => 'cpan'}]},'status' => 404}}
Can't open /CPAN/modules/02packages.details.txt.gz: cannot open file '/CPAN/modules/02packages.details.txt.gz': No such file or directory at /usr/local/lib/perl5/site_perl/5.30.1/Parse/CPAN/Packages/Fast.pm line 97.
File does not exist: /CPAN/authors/00whois.xml at /metacpan-api/bin/../lib/MetaCPAN/Script/Author.pm line 54.
the file you specified with 'path' doesn't exist at /metacpan-api/bin/../lib/MetaCPAN/Script/Permission.pm line 35.

So I see it tries an outgoing connection on Port 873.

anonymous access for rsync

In order to use rsync to anonymously copy data from download.osgeo.org to test.osgeo.net nightly, I would like to have port 873 opened on the firewall.

In this case it is a Firewall Issue because this port is closed by the peripherical firewall on the docker host.

Is there any other way to get those files?

domibay-hugo commented 3 years ago

Once opened the Firewall the Indexation succeeds with:

2021/07/12 14:55:47 I author: Indexing 14137 authors
2021/07/12 14:56:05 I author: done
2021/07/12 14:57:31 I permission: finished indexing 06perms

To run the Tests the api_test cluster must be launched with:

# docker-compose up -d api_test
Creating network "metacpan_traefik-network" with the default driver
Creating network "metacpan_default" with the default driver
Creating network "metacpan_web-network" with the default driver
Creating network "metacpan_elasticsearch" with the default driver
Creating network "metacpan_database" with the default driver
Creating metacpan_elasticsearch_test_1 ... done
Creating metacpan_pgdb_1               ... done
Creating metacpan_pgdb_1               ... 
Creating metacpan_api_test_1           ... done

Then the Tests can be completed as well:

docker-compose exec api_test prove -lvr \
>   t/00_setup.t \
>   t/01_darkpan.t \
>   t/api/controller/cover.t
ok 1 - /metacpan-api/var/t/tmp exists for testing
ok 2 - got ElasticSearch object
# {
#   'Elasticsearch info' => {
#     'cluster_name' => 'elasticsearch',
#     'cluster_uuid' => 'LFW639oFTkSfPs0dZZZA2Q',
#     'name' => 'Tuc',
#     'tagline' => 'You Know, for Search',
#     'version' => {
#       'build_hash' => '5376dca9f70f3abef96a77f4bb22720ace8240fd',
#       'build_snapshot' => bless( do{\(my $o = 0)}, 'JSON::PP::Boolean' ),
#       'build_timestamp' => '2017-07-18T12:17:44Z',
#       'lucene_version' => '5.5.4',
#       'number' => '2.4.6'
#     }
#   }
# }
# [...]
All tests successful.
Files=3, Tests=25, 157 wallclock secs ( 0.04 usr  0.01 sys + 20.10 cusr  1.93 csys = 22.08 CPU)
Result: PASS
oalders commented 3 years ago

Glad to hear you got a working environment. 😃