bartavelle / language-puppet

A library to work with Puppet manifests, test them and eventually replace everything ruby.
BSD 3-Clause "New" or "Revised" License
51 stars 8 forks source link

Exported resource collector #126

Open PierreR opened 9 years ago

PierreR commented 9 years ago

AFAIK the puppetresources command line does not show collected exported resources. There are there on the declaration side but not on the collector side.

Is there a way to enable such a feature ?

bartavelle commented 9 years ago

I don't understand the issue. In this example:

node 'a.b' { 
  @@file { '/tmp/exported':
    ensure => present;
  } 

  File<<| |>>
}

Should output this:

file {
  <@@>/tmp/exported: # "/tmp/x/manifests/site.pp" (line 2, column 12) a.b [Scope []]
    ensure  => present,
    group   => root,
    mode    => 0644,
    path    => /tmp/exported,
    require => Class[::];
}
stage {
  main: # "dummy" (line 1, column 1) a.b [Scope []]
    ;
}
 Exported: 
file {
  <@@>/tmp/exported: # "/tmp/x/manifests/site.pp" (line 2, column 12) a.b [Scope []]
    ensure  => present,
    require => Class[::];
}

Here <@@> means "exported and realized", whereas @@ just means exported.

PierreR commented 9 years ago

I am trying to do something alike

puppetresources -p . -a node1,node2 --hiera ./tests/hiera.yaml --pdbfile ./tests/facts.yaml --checkExported

The first node declares exporting resources and the second node collects them.

bartavelle commented 9 years ago

IIRC it should be seen on the "source" part (the commented part) that this comes from another node.

PierreR commented 7 years ago

@bartavelle I was reviewing this issue to eventually close it. But I still wonder how I can emulate an exported resources faking the puppetdb state with a local pdbfile. Here is a try given both the exported resources from the declaration side (salmaster.prod) and the collect side (pgserver.prod):

facts:
  pgserver.prod:
    role: pgserver
    zone: prod
    environment: middleware_prod
  saltmaster.prod:
    role: saltmaster
    zone: prod
    environment: middleware_prod
resources:
  pgserver.prod:
    data:
      resources:
        - aliases: []
          sourceline: 1
          parameters:
            require: Class[application::profile::postgrest::install]
            db_host: pgserver-cicd.prd.srv.cirb.lan
            db_user: salt
            db_name: saltstack
            port: 3000
          exported: false
          title: postgrest for saltstack
          type: application::profile::postgrest::service
          sourcefile: 'null'
          tags:
          - application::role::pgserver
          - application
          - service
          - node
          - default
          - application::profile::postgrest::service
          - profile
          - role
          - postgrest
          - pgserver
          - application::profile::pgserver
          - class
  salt-master.prod:
    data:
      resources:
        - aliases: []
          sourceline: 1
          parameters:
            require: Class[application::profile::postgrest::install]
            db_host: pgserver-cicd.prd.srv.cirb.lan
            db_user: salt
            db_name: saltstack
            port: 3000
          exported: true
          title: postgrest for saltstack
          type: application::profile::postgrest::service
          sourcefile: 'null'
          tags:
          - application::profile::saltmaster
          - application
          - service
          - node
          - default
          - application::profile::postgrest::service
          - profile
          - role
          - postgrest
          - application::role::saltmaster
          - saltmaster
          - class

I would like to see realized exported resources with: puppetresources -p . -o pgserver.prod --hiera ./tests/hiera.yaml --pdbfile ./tests/facts.yaml but I don't see any mention of them. The manifest/site.pp looks like this:

node default {
    $node_subgroup = pick($::subgroup, ' ')

  if $::role != undef {
    if $::subgroup != undef {
      $role_path =  "${::subgroup}::${::role}"
    } else {
      $role_path = "${::role}"
    }
    include "application::role::${role_path}"
  } else {
    include ::profile::cirb
  }
}

Someone on the pgserver catalog there is this line:

Application::Profile::Postgrest::Service <<| |>>
PierreR commented 7 years ago

I have realized that as soon as I try to add something to resources, I won't even see the declared exported resources when querying the saltmaster.prod node.

I guess there is something wrong in the way I use resources in the pdbfile but I don't see what it is.

bartavelle commented 7 years ago

For me, this saves the resources:

puppetresources -p /tmp/puppet -o a  --pdbfile /tmp/pdb --commitdb --checkExported

And this works when trying to realize them:

puppetresources -p /tmp/puppet -o b  --pdbfile /tmp/pdb

What you posted looks like it should work too. What happens if you move the realization to the node declaration?

PierreR commented 7 years ago

Thanks. Indeed when I try the simple example, it does work. Except that I have unknown in red in the comment which is a bit strange:

[nix-shell:~/projects/cicd/puppet/middleware]$ puppetresources -p . -o b  --hiera tests/hiera.yaml --pdbfile /tmp/pdb
class {
  application::role::test: # SourcePos {sourceName = "./manifests/site.pp", sourceLine = Pos 22, sourceColumn = Pos 5} b [Scope []]
    ;
}
file {
  /tmp/exported: # SourcePos {sourceName = "./manifests/site.pp", sourceLine = Pos 8, sourceColumn = Pos 1} unknown [Scope [import{unknown}{::}]]
    ensure  => present,
    group   => root,
    mode    => 0644,
    path    => /tmp/exported,
    require => Class[::];
}

When I try a real catalog, I am getting errors such as (for instance):

ERROR: (b) The following parameters are unknown: (require) at # SourcePos {sourceName = "./modules/application/manifests/profile/postgrest/service.pp", sourceLine = Pos 6, sourceColumn = Pos 1}

I don't understand where this is coming from.

bartavelle commented 7 years ago

Ugh :(

What do you mean by "real catalog" ?

bartavelle commented 7 years ago

And yeah, it shouldn't be unknown, looks like this code isn't well tested!

PierreR commented 7 years ago

What do you mean by "real catalog" ?

I only have one or two catalog(s) that uses exported resources and the configuration is pretty huge in them involving external puppetlabs modules. When I tried the puppetresources command line on that catalog using pdburl to point to the live puppetdb (or with a faked generated one), I have got errors such as the one I have posted. I suspect there is some bug lying around but it is difficult for me to be more precise at this point. The catalog I have in mind does work fine locally but of course locally there is no state for the exported resources.

PierreR commented 7 years ago

Forget to say it is the node that collects the exported resources that is failing (it is not even clear it fails because of the exported resources) but I have tried several other catalogs against a live puppetdb without problem. Here is another example of such errors:

> puppetresources -p . -o svappxxx661.prd.srv.lan --hiera ./tests/hiera.yaml --pdburl http://puppetdb.prd.srv.lan:8080
ERROR: (svappxxx661.prd.srv.lan) The following parameters are unknown: (before) at # SourcePos {sourceName = "./modules/postgresql/manifests/server/pg_hba_rule.pp", sourceLine = Pos 3, sourceColumn = Pos 1}
bartavelle commented 7 years ago

I suppose there is a problem with the fact that the puppetdb serves relationship parameters as normal parameters, whereas language-puppet handles them differently.