`puppet module search` doesn't give a correct list

[flaf]

Hi,

I have installed puppet-forge-server version commit == 6f1b224a4e666c754876139f3643b22f3515f5e6 and I have this inconsistent behavior explained below.

First, my puppet 4 server uses my local puppet-forge-server, and it seems that the module jippi-php exists:

~# puppet module search php
Notice: Searching http://puppetforge.lss1.backbone.education:8080 ...
NAME        DESCRIPTION                                  AUTHOR       KEYWORDS
jippi-php    Puppet module to manage PHP...     @jippi            php

[...]

But in fact, this module doesn't exist in my local puppetforge and in https://forgeapi.puppetlabs.com too.

~# puppet module install jippi-php
Notice: Preparing to install into /etc/puppetlabs/code/environments/production/modules ...
Notice: Downloading from http://puppetforge.lss1.backbone.education:8080 ...
Error: Could not install 'jippi-php' (latest)
  No releases are available from http://puppetforge.lss1.backbone.education:8080
    Does 'jippi-php' have at least one published release?

Now, I comment this line in the puppet.conf of my puppet 4 server:

#module_repository = http://puppetforge.lss1.backbone.education:8080

Now the puppet 4 server uses https://forgeapi.puppetlabs.com directly:

~# puppet module search php 
Notice: Searching https://forgeapi.puppetlabs.com ...
NAME        DESCRIPTION                                  AUTHOR       KEYWORDS

Now, the module jippi-php is not listed, and this is the normal behavior because this module really doesn't exist.

The problem: why puppet module search php tells me the module jippi-php exists when I use my local puppet-forge-server?

PS: thx for this software very useful for me. This is the first time I notice a problem. ;)

[flaf]

It really seems to me that there is a problem related with the author of a module. For instance, I have no at all the same result with:

# I use the "official" puppet forge.
puppet module search example42 --module_repository=https://forgeapi.puppetlabs.com

# I use my local puppet forge.
puppet module search example42 --module_repository=http://puppetforge.lss1.backbone.education:8080

With the first command I have lines like:

NAME                  DESCRIPTION                   AUTHOR        KEYWORDS
example42-activemq    Puppet module for activemq    @example42    mcollective activemq example42

and with the second I have lines like:

NAME                              DESCRIPTION                   AUTHOR                    KEYWORDS
Alessandro Franceschi-activemq    Puppet module for activemq    @Alessandro Franceschi    activemq example42

[i11]

Hi,

Sorry for the delay... Author field doesn't have any effect on the search as it comes from the presentation layer. However it should be fixed to match the official format.

I'll do some more puppet 4 testing later today to see if I can find something.

[i11]

Right, so probably because of the the way search results are presented jippi-php is actually something else. I'll fix the author field asap.

[i11]

Here is the fix https://github.com/unibet/puppet-forge-server/commit/45589187e669b45a04de29f7b03966a692ccb2b5. We'll add a few more changes before releasing a new version, but you can clone it and test it even now.

Thanks for reporting this!

[flaf]

Hi,

I have just tested and it seems to work well. Now, researches seem to return consistent results, no problem with the author field. Many thx for this quick fix. I will try to follow carefully the next release. Thx again.

François Lafont

[flaf]

Ah sorry, just a question. Now, I can see my private modules no longer. Here is an example with the module flaf-ceph with this metadata.json:

{
  "name": "flaf-ceph",
  "version": "0.1.6",
  "author": "flaf",
  "summary": "Module to manage Ceph clusters and Ceph clients.",
  "license": "Apache-2.0",
  "source": "",
  "project_page": null,
  "issues_url": null,
  "dependencies": [
    {"name":"puppetlabs-stdlib","version_requirement":">= 4.6.0"},
    {"name":"flaf-repository","version_requirement":">= 0.1.2"}
  ]
}

Now, I have this:

~# puppet module search flaf
Notice: Searching http://puppetforge.lss1.backbone.education:8080 ...
No results found for 'flaf'.

~# puppet module search ceph
Notice: Searching http://puppetforge.lss1.backbone.education:8080 ...
NAME              DESCRIPTION                   AUTHOR        KEYWORDS
spjmurray-ceph    Ceph orchestration            @spjmurray    ceph
eNovance-ceph     A puppet module to install... @eNovance     ceph
datacentred-ceph  Module for managing ceph      @datacentred
jeroenmoors-ceph  UNKNOWN                       @jeroenmoors  debian ubuntu ceph

Is there still a problem or maybe something is lacking in my metadata.json to be visible during a research?

[i11]

The fix doesn't have any effect on the search results. It just changed the way the author was presented.

What's the content of your private module directory? Local search is file name based, so for example if the module file doesn't contain flaf in its name it won't be included in the results.

Could you please try to test both released 1.8.0 and the latest commit?

[flaf]

My private modules are not so private and all is here https://github.com/flaf/miscellaneous/tree/master/puppet/modules (where you can see the ceph module for instance).

It seems to me there is a little regression somewhere because:

• puppet module search flaf => empty result with the "last" version.
• puppet module search flaf => I can see all my private modules with the 1.8.0 version (ie with commit 6f1b224a4e666c754876139f3643b22f3515f5e6)

[flaf]

Ah, new information: I have errors in the log (in the puppetforge) during a research:

[2015-11-23 23:03:52] ERROR  Failed reading metadata from /var/lib/puppetforge/modules/flaf-unix_accounts-0.2.10.tar.gz
[...]

The puppetforge server is install exactly with a personal puppet module here https://github.com/flaf/miscellaneous/tree/master/puppet/modules/puppetforge

[flaf]

Another log

Error reading from module archive /var/lib/puppetforge/modules/flaf-puppetforge-0.2.9.tar.gz: Given name (?-mix:^([^\/]+\/)?(metadata\.json|README\.md)$) not found in /var/lib/puppetforge/modules/flaf-puppetforge-0.2.9.tar.gz
[2015-11-23 23:07:46] ERROR  Failed reading metadata from /var/lib/puppetforge/modules/flaf-puppetforge-0.2.9.tar.gz

[i11]

Doh.. that's probably caused by https://github.com/unibet/puppet-forge-server/pull/34. I've created a new branch https://github.com/unibet/puppet-forge-server/tree/author-fix (latest commit: https://github.com/unibet/puppet-forge-server/commit/54df58f83d415d3b98eeb7a1bc8b4e1cbb841d90) just with author field fixes. It would be great if you could test that one instead.

[flaf]

I have tried this version (commit 54df58f83d415d3b98eeb7a1bc8b4e1cbb841d90 branch author-fix) but I have the same problem.

 ERROR  Failed reading metadata from /var/lib/puppetforge/modules/flaf-repository-0.1.1.tar.gz
Error reading from module archive /var/lib/puppetforge/modules/flaf-repository-0.1.4.tar.gz: Given name (?-mix:^([^\/]+\/)?metadata\.json$) not found in /var/lib/puppetforge/modules/flaf-repository-0.1.4.tar.gz

With this version, I don't know why but I have a problem to start the application. I have installed the application in /opt/puppetforge-server where Unix rights of the files are root:root 755/644. And I start the application with another account puppetforge which can't at all write in /opt/puppetforge-server. And if I want to start the application I have this error:

There was an error while trying to write to Gemfile.lock. It is likely that 
you need to allow write permissions for the file at path: 
/opt/puppetforge-server/Gemfile.lock

To be able to test this version, I had changed Unix rights in the install directory but it was not very clean. It will be possible to start the application without need to write in the install directory?

[i11]

Sorry for disappearing like that. Had a bit too much on my head. Is this still an issue? I'll be releasing the master after https://github.com/unibet/puppet-forge-server/pull/35 is sorted out and merged.

[flaf]

Hi, No problem ;) I have tried the version ac9618c0ef564c6831158b2bf43c25f3e9148f93 (ie the last commit) and indeed the puppet module search ... seems to work well except that my modules (not the modules of the official puppet forge) are not found. Typical example with puppet module search flaf I have no result and in /var/log/puppetforge/server.log I have for instance:

ERROR  Failed reading metadata from /var/lib/puppetforge/modules/flaf-unix_accounts-0.2.20.tar.gz

I will recheck. Maybe the problem is in my side... Is there changes in the metadata.json now in this new version of puppet-forge-server?

[flaf]

I think there is still a problem but different. With exactly the same installation and the same personal modules:

• With this commit 6f1b224a4e666c754876139f3643b22f3515f5e6 (old commit), I have the problem described in this present issue but my personal modules are visible.
• But with the last commit (ac9618c0ef564c6831158b2bf43c25f3e9148f93), searching looks fine except for my personal modules in my puppet-forge-server which are completely invisible. I have this in the log:

Failed reading metadata from /var/lib/puppetforge/modules/flaf-basic_packages-0.1.2.tar.gz

So, according to me, there is a little regression somewhere but I don't know where. Maybe it can help, here is an example of personal module which triggers the error message above: http://francois-lafont.ac-versailles.fr/misc/flaf-basic_packages-0.1.2.tar.gz

Regards

[i11]

Ok, so the posted module really helped to understand the issue. It contains original directory tree (var/lib/puppetforge/modules/) where it's expected to have module files at the root of the archive.

How do you build modules? I haven't seen puppet module build produce such archives.

[flaf]

Hi, sorry for my late answer. Indeed, I use a manual tar command to build my tar.gz module and I don't use puppet module build (the specific reason is I want to keep the .git/ directory in my modules to be able to commit/push directly in my puppetserver). So I use a specific script to build my tar.gz

Indeed, you are right, my method doesn't match exactly with the puppet module build command. So I have fixed my script (especially the tar command) and now the original directory are in the archives no longer. Then I have tested the commit ab01f3376be081a426798b4333aed0ada920f637 (ie version 1.9.0) and now all seems to be ok and the researches puppet module search xxxx seem to be ok.

It's curious but my personal building script was wrong since the beginning and it worked well with the version 1.8.0 (it was probably by fluke).

Anyway, now all is ok for me. Thx a lot Ilja for the updates and for the help too. ;) Regards.

[i11]

No problem. Glad I coud help. Somewhere along the way I've fixed how module files are being searched for meatada.json and that revealed the issue.

I'm closing the issue.