Unable to get environment started

[bleargh45]

Ok... feeling like either I'm missing something, or that this Docker setup doesn't actually run (which could entirely be the piece I'm missing).

Tried following the instructions to set up metacpan-docker, but after running bin/metacpan-docker localapi up, although all of the containers appear to be up, neither the "web" nor the "api" container respond properly.

"web" container displays an error page indicating:

Couldn't render template "home.html": file error - preprocess.html: not found

"api" container just does not respond via HTTP at all. Examining the logs for "api", though, and I do see:

Can't load application from file "/metacpan-api/bin/api.pl": DBI connect('dbname=minion_queue;host=pgdb','metacpan',...) failed: fe_sendauth: no password supplied at /usr/local/lib/perl5/site_perl/5.30.1/Mojo/Pg.pm line 91

Tried hacking the src/metacpan-api/metacpan_server.conf file to include a username+password, at which point I then get:

Can't load application from file "/metacpan-api/bin/api.pl": DBI connect('dbname=minion_queue;host=pgdb;port=5432','metacpan',...) failed: FATAL:  password authentication failed for user "metacpan" at /usr/local/lib/perl5/site_perl/5.30.1/Mojo/Pg.pm line 91.

So... it won't connect without a password, but it also then won't log in with the POSTGRES_PASSWORD that is listed in docker-compose.yml

Would someone be kind enough to point me in the direction of what I'm missing, so that I can get a functioning environment set up to hack on MetaCPAN locally? I'm primarily interested in trying to hack on metacpan-api, and then possibly metacpan-web.

[oalders]

Hi @bleargh45,

This should all be working. See, for example, the CircleCI config for metacpan-api, which uses docker-compose: https://github.com/metacpan/metacpan-api/blob/master/.circleci/config.yml That's a currently working example of how to get the -api tests up and running.

This repo has the following CI config that should also be working: https://github.com/metacpan/metacpan-docker/blob/master/.circleci/config.yml

For your work, did you run ./bin/metacpan-docker init before getting started?

What happens if you run docker-compose up api?

[bleargh45]

Yes, I did run bin/metacpan-docker init before getting started.

However, after running docker-compose up api, I continue to get the same error;

api_1                     | wait-for-it.sh: waiting 15 seconds for pgdb:5432
api_1                     | wait-for-it.sh: pgdb:5432 is available after 0 seconds
api_1                     | Server available at http://127.0.0.1:5000
api_1                     | Can't load application from file "/metacpan-api/bin/api.pl": DBI connect('dbname=minion_queue;host=pgdb','metacpan',...) failed: fe_sendauth: no password supplied at /usr/local/lib/perl5/site_perl/5.30.1/Mojo/Pg.pm line 91.
api_1                     | Compilation failed in require at (eval 91) line 1.

after which point I can try to connect to :5000 but it just spins/stalls (which I am presuming is because while Mojo has bound to the port, there is no App running).

Am going to try blowing away all of the containers/images, re-initializing with bin/metacpan-docker init again, and see if that helps flush something out that may be funky on my end.

[bleargh45]

Hmm. Nope. Even after shutting down all running containers, and doing both a docker container prune and a docker image prune, restarting things with docker-compose up api (and watching it rebuild things) still presents me with the same DBI error.

Using docker-compose up api_test, however, gets the App started without that error. Doesn't allow me to connect to it from my laptop, but I am presuming that is because this is a container for running tests in and so the port isn't exposed (e.g. the inability to connect to this container from my laptop was not unexpected).

So... now for me to go take a look and see what the difference is between the startup of the api and api_test setups...

[oalders]

I've just been running tests using Docker lately, so it's quite possible you've discovered a real problem with the setup here. The live deployment doesn't (yet) use docker-compose. @mickeyn have you been using api lately or just api_test?

[bleargh45]

So... if I set the api container in docker-compose.yml to use localapi_test.env instead of just localapi.env, that gets me an env that starts up with docker-compose up api, and which I can reach from my laptop with a browser.

Doesn't (yet) have anything in the index, but I expect that is because I have not yet run bin/metacpan-docker localapi exec api index-cpan.sh to populate the indexes with any data.

[ssoriche]

The localapi_test.env is using the src/metacpan-api/metacpan_server_testing.conf which doesn't specify minion_dsn so won't throw the same error.

The setup for PostgreSQL user metacpan does not have a password, yet there is one being set in the environment variable. A quick update to pg/docker-entrypoint-initdb.d/100-roles.sql to add the password should fix the problem. I might be able to test this out later tonight.

[bleargh45]

Well, have taken a few other stabs at this, including multiple "blow everything away, and start all over again from scratch", but continue to be unable to get a functioning environment running.

Appreciate the comments and discussion above, but so far, no luck.

[oalders]

@bleargh45 which host OS are you running?

[bleargh45]

Have tried both on Fedora-33, as well as on OS X "Catalina", and have been unable to get it to work properly on either.

[oalders]

@bleargh45 thanks. Just curious what your immediate priorities are. It's possible to index releases without the queue, so if you don't need the queue running right now, you could just comment out the Minion app logic for the time being so that you're not blocked on it.

[bleargh45]

My goal was to be able to hack on the API + Web interfaces, to play with better notifications for "this module (or distribution) has been marked as Deprecated".

Have a few Dists like that myself, and having seen the extra notifications that are in place for things like ADOPTME, I thought it would be nice to add in a nicer display for "this is Deprecated, might want to avoid".

[oalders]

That's helpful. You should be able to bypass the queue entirely for this work. You may not even need to run the API at all. You might be able to get away with just metacpan-web. If that's the case, you could either start with just firing up that Docker image or even just installing the deps and running it via Plackup.

Probably I would fire up metacpan-web and play around with that. If the live API responses don't suit your needs, then let us know and together we can look at what's involved in tweaking the API. I think that's the path of least resistance.

[bleargh45]

Ok... think I've got this working, with some help from @ssoriche.

Have submitted two PR's to help address the issues that I encountered:

• 82; Fix metacpan_web config

• 83; Fix PGDB authentication

[oalders]

Excellent! I'll close this ticket. Let me know if we need to re-open it.