ProgrammeVitam / vitam

Digital Archives Management System developped by French government/Programme interministériel archives numériques ; core system.
CeCILL Free Software License Agreement v2.1
122 stars 41 forks source link

Erreur Keystore vitam-storage sur deploiement medium #12

Closed bxaxa closed 4 years ago

bxaxa commented 5 years ago

Bonjour, j'ai réussi un déploiement xsmall, small et je tente un déploiement medium, qui semble ne pas spécialement différer au niveau de la procédure d'installation, mais j'ai un problème au déploiement sur 15 VMS

TASK [vitam : Check if vitam-storage is listening on service port 9102] *******************************************************************************************************************************************************************
fatal: [medium-app-vm-2]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 10.65.56.25:9102"}

NO MORE HOSTS LEFT ************************************************************************************************************************************************************************************************************************

PLAY RECAP ********************************************************************************************************************************************************************************************************************************
medium-access-vm-1         : ok=35   changed=17   unreachable=0    failed=0    skipped=19   rescued=0    ignored=0
medium-admin-vm-1          : ok=126  changed=66   unreachable=0    failed=0    skipped=43   rescued=0    ignored=2
medium-admin-vm-2          : ok=91   changed=51   unreachable=0    failed=0    skipped=30   rescued=0    ignored=1
medium-admin-vm-3          : ok=91   changed=51   unreachable=0    failed=0    skipped=30   rescued=0    ignored=1
medium-app-vm-1            : ok=35   changed=17   unreachable=0    failed=0    skipped=15   rescued=0    ignored=0
medium-app-vm-2            : ok=53   changed=27   unreachable=0    failed=1    skipped=29   rescued=0    ignored=0
medium-app-vm-3            : ok=35   changed=17   unreachable=0    failed=0    skipped=15   rescued=0    ignored=0
medium-app-vm-4            : ok=35   changed=17   unreachable=0    failed=0    skipped=15   rescued=0    ignored=0
medium-data-vm-1           : ok=122  changed=69   unreachable=0    failed=0    skipped=55   rescued=0    ignored=1
medium-data-vm-2           : ok=110  changed=61   unreachable=0    failed=0    skipped=63   rescued=0    ignored=1
medium-data-vm-3           : ok=125  changed=72   unreachable=0    failed=0    skipped=48   rescued=0    ignored=1
medium-external-vm-1       : ok=35   changed=17   unreachable=0    failed=0    skipped=15   rescued=0    ignored=0
medium-storage-vm-1        : ok=118  changed=67   unreachable=0    failed=0    skipped=80   rescued=0    ignored=0
medium-storage-vm-2        : ok=80   changed=44   unreachable=0    failed=0    skipped=76   rescued=0    ignored=0
medium-storage-vm-3        : ok=95   changed=55   unreachable=0    failed=0    skipped=61   rescued=0    ignored=0

et quand je regarde les logs de vitam-storage sur la medium-app-vm-2

cat storage.2019-08-01.0.log

2019-08-01 06:53:38,127 [[main]] [] ERROR org.elasticsearch.metrics.ElasticsearchReporter - Caller+1     at org.elasticsearch.metrics.ElasticsearchReporter.<init>(ElasticsearchReporter.java:264) : Error adding metrics template to elasticsearch: Bad Request/{}400
2019-08-01 06:53:38,380 [[main]] [] ERROR org.elasticsearch.metrics.ElasticsearchReporter - Caller+1     at org.elasticsearch.metrics.ElasticsearchReporter.<init>(ElasticsearchReporter.java:264) : Error adding metrics template to elasticsearch: Bad Request/{}400
2019-08-01 06:53:38,397 [[main]] [] ERROR org.elasticsearch.metrics.ElasticsearchReporter - Caller+1     at org.elasticsearch.metrics.ElasticsearchReporter.<init>(ElasticsearchReporter.java:264) : Error adding metrics template to elasticsearch: Bad Request/{}400
2019-08-01 06:53:39,052 [[main]] [] ERROR fr.gouv.vitam.storage.engine.server.rest.StorageResource - Caller+1    at fr.gouv.vitam.storage.engine.server.rest.StorageResource.<init>(StorageResource.java:174) : [vitam-TEST-medium-app-vm-2:storage:142004237] unable to instantiate TimeStampGenerator java.io.FileNotFoundException: File not found: keystore_secure-storage.p12

J'ai utilisé la même procédure que pour la version small (3 vms), j'ai juste utilisé le hosts.medium et ajouté plus de vms

croftophile commented 5 years ago

Bonjour, Avez-vous bien "(re)joué" la PKI (ou réalisé les actions similaires avec votre propre PKI) avant de lancer le déploiement sur cet environnement "medium" ?

bxaxa commented 5 years ago

J'ai rejoué la PKI, voici exactement ce que j'ai fait. Je fait exactemement la même chose pour la version small et l'installation fonctionne nickel.

cd vitam/deployment/
cp environments/hosts.example.medium environments/hosts.medium
pki/scripts/generate_ca.sh
pki/scripts/generate_certs.sh environments/hosts.medium
./generate_stores.sh
nroselier commented 5 years ago

Bonjour,

Pouvez vous vérifier que vous avez bien ce certificat dans votre arborescence ?

environments/certs/timestamping/vitam/secure-storage.crt

Explications sur le fonctionnement au déploiement :

Merci d'avance, Cordialement

bxaxa commented 5 years ago

Bonjour, le keystore est bien généré sur la machine de deploiement mais il n'est jamais copié sur la machine vitam.

TASK [vitam : Copy timestamp keystore] ***********************************************************************************************************************************************************************************************
task path: /home/amapi/vitam/vitam/deployment/ansible-vitam/roles/vitam/tasks/main.yml:266
 [WARNING]: conditional statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: ( vitam_timestamp_usage is defined ) and ( {{ lookup('pipe', 'test -f
{{inventory_dir}}/keystores/timestamping/keystore_{{vitam_timestamp_usage}}.p12 || echo nofile') == "" }} )

skipping: [medium-app-vm-2] => {
    "changed": false,
    "skip_reason": "Conditional result was False"

Apparement il y a un soucis dans la condition.

ps: la commande lookup n'était pas installée sur mon poste de deploiement. Je reteste

ce n'était pas ca

nroselier commented 5 years ago

Bonjour,

Nous pensons que l'erreur peut venir de la commande test -f {{inventory_dir}}/keystores/timestamping/keystore_{{vitam_timestamp_usage}}.p12

Pouvez-vous tester cette commande sur votre machine ?

Le playbook vitam.yml contient-il bien la variable vitam_timestamp_usage: secure-storage (le playbbok a-t-il été modifié) ?

Cordialement, Nicolas Roselier.

bxaxa commented 5 years ago

Le playbook n'a pas été modifié (je fait un git clone à chaque install).

La variable semble bien définie:

- hosts: hosts-storage-engine
  any_errors_fatal: true
  roles:
    - vitam
    - timers
    - storage
  vars:
    vitam_struct: "{{ vitam.storageengine }}"
    vitam_certificate_client_type: "storage"
    vitam_timestamp_usage: secure-storage
    timers_list: "{{ vitam_timers.storage }}"

Je vais tester la commande

bxaxa commented 5 years ago

Je viens de voir que le p12 n'est pas généré, et pour cause. Quand je regarde le repertoire hosts, il me manque pas mal de hosts.

ls environments/certs/server/hosts/
localhost  medium-access-vm-1  medium-external-vm-1  medium-storage-vm-1
bxaxa commented 5 years ago

Je reposte mes logs il y a des trucs en trop

bxaxa commented 5 years ago

COMMANDE 1

pki/scripts/generate_ca.sh

[INFO] [generate_ca.sh: main] Lancement de la procédure de création des CA
[INFO] [generate_ca.sh: main] ==============================================
[INFO] [generate_ca.sh: initVault] Réinitialisation du fichier /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/vault-ca.yml
Decryption successful
Encryption successful
[INFO] [generate_ca.sh: main] Création de CA root pour server...
Decryption successful
Encryption successful
[INFO] [generate_ca.sh: generate_ca_root] Create CA request...
Generating a RSA private key
...........................................++++
......................................................................................................................................................................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/ca/server/ca-root.key'
-----
[INFO] [generate_ca.sh: generate_ca_root] Create CA certificate...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/ca-config
Can't open ./pki/config/server/index.txt.attr for reading, No such file or directory
140572156289472:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/server/index.txt.attr','r')
140572156289472:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
organizationalUnitName:ASN.1 12:'authorities'
commonName            :ASN.1 12:'ca_root_server'
Certificate is to be certified until Jul 30 11:31:21 2029 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
[INFO] [generate_ca.sh: main] Création de la CA intermediate pour server...
Decryption successful
Encryption successful
[INFO] [generate_ca.sh: generate_ca_interm] Generate intermediate request...
Generating a RSA private key
.....................................................++++
..........................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/ca/server/ca-intermediate.key'
-----
[INFO] [generate_ca.sh: generate_ca_interm] Sign...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/ca-config
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
organizationalUnitName:ASN.1 12:'authorities'
commonName            :ASN.1 12:'ca_intermediate_server'
Certificate is to be certified until Jul 30 11:31:23 2029 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
[INFO] [generate_ca.sh: main] ----------------------------------------------
[INFO] [generate_ca.sh: main] Création de CA root pour client-external...
Decryption successful
Encryption successful
[INFO] [generate_ca.sh: generate_ca_root] Create CA request...
Generating a RSA private key
.........++++
..........++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/ca/client-external/ca-root.key'
-----
[INFO] [generate_ca.sh: generate_ca_root] Create CA certificate...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/ca-config
Can't open ./pki/config/client-external/index.txt.attr for reading, No such file or directory
140541302665664:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/client-external/index.txt.attr','r')
140541302665664:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
organizationalUnitName:ASN.1 12:'authorities'
commonName            :ASN.1 12:'ca_root_client-external'
Certificate is to be certified until Jul 30 11:31:24 2029 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
[INFO] [generate_ca.sh: main] Création de la CA intermediate pour client-external...
Decryption successful
Encryption successful
[INFO] [generate_ca.sh: generate_ca_interm] Generate intermediate request...
Generating a RSA private key
..............................................................................++++
.........................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/ca/client-external/ca-intermediate.key'
-----
[INFO] [generate_ca.sh: generate_ca_interm] Sign...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/ca-config
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
organizationalUnitName:ASN.1 12:'authorities'
commonName            :ASN.1 12:'ca_intermediate_client-external'
Certificate is to be certified until Jul 30 11:31:26 2029 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
[INFO] [generate_ca.sh: main] ----------------------------------------------
[INFO] [generate_ca.sh: main] Création de CA root pour client-storage...
Decryption successful
Encryption successful
[INFO] [generate_ca.sh: generate_ca_root] Create CA request...
Generating a RSA private key
...................................................................................................................++++
......................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/ca/client-storage/ca-root.key'
-----
[INFO] [generate_ca.sh: generate_ca_root] Create CA certificate...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/ca-config
Can't open ./pki/config/client-storage/index.txt.attr for reading, No such file or directory
139853835866560:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/client-storage/index.txt.attr','r')
139853835866560:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
organizationalUnitName:ASN.1 12:'authorities'
commonName            :ASN.1 12:'ca_root_client-storage'
Certificate is to be certified until Jul 30 11:31:29 2029 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
[INFO] [generate_ca.sh: main] Création de la CA intermediate pour client-storage...
Decryption successful
Encryption successful
[INFO] [generate_ca.sh: generate_ca_interm] Generate intermediate request...
Generating a RSA private key
......................................................................................................................................................................................................................................................................................................................................++++
....................................................................................................................................................................................................................................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/ca/client-storage/ca-intermediate.key'
-----
[INFO] [generate_ca.sh: generate_ca_interm] Sign...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/ca-config
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
organizationalUnitName:ASN.1 12:'authorities'
commonName            :ASN.1 12:'ca_intermediate_client-storage'
Certificate is to be certified until Jul 30 11:31:33 2029 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
[INFO] [generate_ca.sh: main] ----------------------------------------------
[INFO] [generate_ca.sh: main] Création de CA root pour timestamping...
Decryption successful
Encryption successful
[INFO] [generate_ca.sh: generate_ca_root] Create CA request...
Generating a RSA private key
............................................................................................................................................................................................................++++
....++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/ca/timestamping/ca-root.key'
-----
[INFO] [generate_ca.sh: generate_ca_root] Create CA certificate...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/ca-config
Can't open ./pki/config/timestamping/index.txt.attr for reading, No such file or directory
139825049752000:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/timestamping/index.txt.attr','r')
139825049752000:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
organizationalUnitName:ASN.1 12:'authorities'
commonName            :ASN.1 12:'ca_root_timestamping'
Certificate is to be certified until Jul 30 11:31:35 2029 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
[INFO] [generate_ca.sh: main] Création de la CA intermediate pour timestamping...
Decryption successful
Encryption successful
[INFO] [generate_ca.sh: generate_ca_interm] Generate intermediate request...
Generating a RSA private key
...............................................................................................................................++++
.............................................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/ca/timestamping/ca-intermediate.key'
-----
[INFO] [generate_ca.sh: generate_ca_interm] Sign...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/ca-config
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
organizationalUnitName:ASN.1 12:'authorities'
commonName            :ASN.1 12:'ca_intermediate_timestamping'
Certificate is to be certified until Jul 30 11:31:38 2029 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated
[INFO] [generate_ca.sh: main] ----------------------------------------------
[INFO] [generate_ca.sh: main] ==============================================
[INFO] [generate_ca.sh: main] Fin de la procédure de création des CA

COMMANDE 2

pki/scripts/generate_certs.sh environments/hosts.medium

[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[INFO] [generate_certs.sh: initVault] Réinitialisation du fichier /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/vault-certs.yml
Decryption successful
Encryption successful
[INFO] [generate_certs.sh: main] Recopie des clés publiques des CA
[INFO] [generate_certs.sh: copyCAFromPki] Copie de la CA (root + intermediate) de client-external
[INFO] [generate_certs.sh: copyCAFromPki] Copie de la CA (root + intermediate) de client-storage
[INFO] [generate_certs.sh: copyCAFromPki] Copie de la CA (root + intermediate) de server
[INFO] [generate_certs.sh: copyCAFromPki] Copie de la CA (root + intermediate) de timestamping
[INFO] [generate_certs.sh: main] Génération des certificats serveurs
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[INFO] [generate_certs.sh: generateHostCertificate] Création du certificat server pour ingest-external hébergé sur medium-external-vm-1...
[INFO] [generate_certs.sh: generateHostCertificate] Generation de la clé...
Generating a RSA private key
........................................................................................++++
...........................................................................................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/server/hosts/medium-external-vm-1/ingest-external.key'
-----
[INFO] [generate_certs.sh: generateHostCertificate] Generation du certificat signé avec CA server...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/server/index.txt.attr for reading, No such file or directory
139997061738944:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/server/index.txt.attr','r')
139997061738944:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'ingest-external.service.local.consul'
Certificate is to be certified until Aug  1 11:31:57 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[INFO] [generate_certs.sh: generateHostCertificate] Création du certificat server pour access-external hébergé sur medium-external-vm-1...
[INFO] [generate_certs.sh: generateHostCertificate] Generation de la clé...
Generating a RSA private key
........................................++++
.................................................................................................................................................................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/server/hosts/medium-external-vm-1/access-external.key'
-----
[INFO] [generate_certs.sh: generateHostCertificate] Generation du certificat signé avec CA server...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/server/index.txt.attr for reading, No such file or directory
140196209906112:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/server/index.txt.attr','r')
140196209906112:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'access-external.service.local.consul'
Certificate is to be certified until Aug  1 11:32:02 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[INFO] [generate_certs.sh: generateHostCertificate] Création du certificat server pour offer hébergé sur medium-storage-vm-1...
[INFO] [generate_certs.sh: generateHostCertificate] Generation de la clé...
Generating a RSA private key
...............................................................................................................................................................++++
....................................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/server/hosts/medium-storage-vm-1/offer.key'
-----
[INFO] [generate_certs.sh: generateHostCertificate] Generation du certificat signé avec CA server...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/server/index.txt.attr for reading, No such file or directory
140245654901184:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/server/index.txt.attr','r')
140245654901184:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'offer.service.local.consul'
Certificate is to be certified until Aug  1 11:32:06 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
 [WARNING]: No hosts matched, nothing to do
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed in version 2.10. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[INFO] [generate_certs.sh: generateHostCertificate] Création du certificat server pour ihm-demo hébergé sur medium-access-vm-1...
[INFO] [generate_certs.sh: generateHostCertificate] Generation de la clé...
Generating a RSA private key
.................................................................................................................................++++
....++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/server/hosts/medium-access-vm-1/ihm-demo.key'
-----
[INFO] [generate_certs.sh: generateHostCertificate] Generation du certificat signé avec CA server...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/server/index.txt.attr for reading, No such file or directory
140150179918272:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/server/index.txt.attr','r')
140150179918272:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'ihm-demo.service.local.consul'
Certificate is to be certified until Aug  1 11:32:12 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[INFO] [generate_certs.sh: main] Génération des certificats timestamping
[INFO] [generate_certs.sh: generateTimestampCertificate] Création du certificat timestamping pour usage secure-logbook
[INFO] [generate_certs.sh: generateTimestampCertificate] Generation de la clé...
Generating a RSA private key
...++++
........................................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/timestamping/vitam/secure-logbook.key'
-----
[INFO] [generate_certs.sh: generateTimestampCertificate] Generation du certificat signé avec CA timestamping...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/timestamping/index.txt.attr for reading, No such file or directory
140395629171136:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/timestamping/index.txt.attr','r')
140395629171136:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'secure-logbook'
Certificate is to be certified until Aug  1 11:32:16 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[INFO] [generate_certs.sh: generateTimestampCertificate] Création du certificat timestamping pour usage secure-storage
[INFO] [generate_certs.sh: generateTimestampCertificate] Generation de la clé...
Generating a RSA private key
....................................................................................................................................................................................++++
..................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/timestamping/vitam/secure-storage.key'
-----
[INFO] [generate_certs.sh: generateTimestampCertificate] Generation du certificat signé avec CA timestamping...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/timestamping/index.txt.attr for reading, No such file or directory
140153497362880:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/timestamping/index.txt.attr','r')
140153497362880:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'secure-storage'
Certificate is to be certified until Aug  1 11:32:19 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[INFO] [generate_certs.sh: main] Génération des certificats clients
[INFO] [generate_certs.sh: generateClientCertificate] Création du certificat client pour ihm-demo
[INFO] [generate_certs.sh: generateClientCertificate] Generation de la clé...
Generating a RSA private key
...............................++++
......................................................................................................................................................................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/client-external/clients/ihm-demo/ihm-demo.key'
-----
[INFO] [generate_certs.sh: generateClientCertificate] Generation du certificat signé avec client-external...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/client-external/index.txt.attr for reading, No such file or directory
140620239061440:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/client-external/index.txt.attr','r')
140620239061440:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'ihm-demo'
Certificate is to be certified until Aug  1 11:32:23 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[INFO] [generate_certs.sh: generateClientCertificate] Création du certificat client pour gatling
[INFO] [generate_certs.sh: generateClientCertificate] Generation de la clé...
Generating a RSA private key
.....................++++
..............++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/client-external/clients/gatling/gatling.key'
-----
[INFO] [generate_certs.sh: generateClientCertificate] Generation du certificat signé avec client-external...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/client-external/index.txt.attr for reading, No such file or directory
140291121689024:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/client-external/index.txt.attr','r')
140291121689024:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'gatling'
Certificate is to be certified until Aug  1 11:32:26 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[INFO] [generate_certs.sh: generateClientCertificate] Création du certificat client pour vitam-admin-int
[INFO] [generate_certs.sh: generateClientCertificate] Generation de la clé...
Generating a RSA private key
............................................++++
.............................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/client-external/clients/vitam-admin-int/vitam-admin-int.key'
-----
[INFO] [generate_certs.sh: generateClientCertificate] Generation du certificat signé avec client-external...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/client-external/index.txt.attr for reading, No such file or directory
140567175590336:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/client-external/index.txt.attr','r')
140567175590336:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'vitam-admin-int'
Certificate is to be certified until Aug  1 11:32:29 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[INFO] [generate_certs.sh: generateClientCertificate] Création du certificat client pour ihm-recette
[INFO] [generate_certs.sh: generateClientCertificate] Generation de la clé...
Generating a RSA private key
.............................................................................................++++
.........................................................................................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/client-external/clients/ihm-recette/ihm-recette.key'
-----
[INFO] [generate_certs.sh: generateClientCertificate] Generation du certificat signé avec client-external...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/client-external/index.txt.attr for reading, No such file or directory
140144009118144:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/client-external/index.txt.attr','r')
140144009118144:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'ihm-recette'
Certificate is to be certified until Aug  1 11:32:33 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[INFO] [generate_certs.sh: generateClientCertificate] Création du certificat client pour reverse
[INFO] [generate_certs.sh: generateClientCertificate] Generation de la clé...
Generating a RSA private key
......................................................................................................................................................................................................................++++
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/client-external/clients/reverse/reverse.key'
-----
[INFO] [generate_certs.sh: generateClientCertificate] Generation du certificat signé avec client-external...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/client-external/index.txt.attr for reading, No such file or directory
139640118989248:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/client-external/index.txt.attr','r')
139640118989248:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'reverse'
Certificate is to be certified until Aug  1 11:32:39 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[INFO] [generate_certs.sh: main] Génération des certificats storage
[INFO] [generate_certs.sh: generateClientCertificate] Création du certificat client pour storage
[INFO] [generate_certs.sh: generateClientCertificate] Generation de la clé...
Generating a RSA private key
..........................................................................................................................++++
......................................................................................................................++++
writing new private key to '/home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/client-storage/clients/storage/storage.key'
-----
[INFO] [generate_certs.sh: generateClientCertificate] Generation du certificat signé avec client-storage...
Using configuration from /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/pki/config/crt-config
Can't open ./pki/config/client-storage/index.txt.attr for reading, No such file or directory
139962203419072:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:72:fopen('./pki/config/client-storage/index.txt.attr','r')
139962203419072:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:79:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'fr'
stateOrProvinceName   :ASN.1 12:'idf'
localityName          :ASN.1 12:'paris'
organizationName      :ASN.1 12:'vitam'
commonName            :ASN.1 12:'storage'
Certificate is to be certified until Aug  1 11:32:43 2022 GMT (1095 days)

Write out database with 1 new entries
Data Base Updated
Decryption successful
Encryption successful
[INFO] [generate_certs.sh: main] Fin de script

COMMANDE 3

 ./generate_stores.sh

[INFO] [generate_stores.sh: main] -------------------------------------------
[INFO] [generate_stores.sh: main] Creation du keystore de access-external pour le serveur localhost
[INFO] [generate_stores.sh: generateHostKeystore] Génération du p12
[INFO] [generate_stores.sh: generateHostKeystore] Génération du jks
Importing keystore /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/server/hosts/localhost/access-external.p12 to /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/keystores/server/localhost/keystore_access-external.jks...
Entry for alias access-external successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/keystores/server/localhost/keystore_access-external.jks -destkeystore /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/keystores/server/localhost/keystore_access-external.jks -deststoretype pkcs12".
[INFO] [generate_stores.sh: generateHostKeystore] Suppression du p12
[INFO] [generate_stores.sh: main] -------------------------------------------
[INFO] [generate_stores.sh: main] Creation du keystore de ihm-demo pour le serveur localhost
[INFO] [generate_stores.sh: generateHostKeystore] Génération du p12
[INFO] [generate_stores.sh: generateHostKeystore] Génération du jks
Importing keystore /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/certs/server/hosts/localhost/ihm-demo.p12 to /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/keystores/server/localhost/keystore_ihm-demo.jks...
Entry for alias ihm-demo successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/keystores/server/localhost/keystore_ihm-demo.jks -destkeystore /home/amapi/vvitam/vitam-at-scaleway/vitam/deployment/environments/keystores/server/localhost/keystore_ihm-demo.jks -deststoretype pkcs12".
[INFO] [generate_stores.sh: generateHostKeystore] Suppression du p12
[INFO] [generate_stores.sh: main] -------------------------------------------
[INFO] [generate_stores.sh: main] Creation du keystore de ihm-recette pour le serveur localhost
[ERROR] [generate_stores.sh: getComponentPassphrase] Error while retrieving the key: server_ihm-recette_key

COMMANDE 4

ls environments/certs/server/hosts

localhost  medium-access-vm-1  medium-external-vm-1  medium-storage-vm-1
nroselier commented 5 years ago

Testez vous une version 2.1.6 (R9.6) sur Debian ? Un bug lié à l'utilisation de awk sur Debian nous a déjà été remonté en juin dernier sur la version 2.1.6. Un fix avait été apporté en R9.8 (2.1.8). Cf. https://github.com/ProgrammeVitam/vitam/commit/29e807f03e15f3ffcf28aaee69ae1de40f6567d1

Cordialement

bxaxa commented 5 years ago

Je suis sur vitam 2.6.3-1

git status On branch master_2.6.x Your branch is up to date with 'origin/master_2.6.x'.

bxaxa commented 5 years ago

Définitivement, pour 15 Vms, il y a un soucis avec la PKI. le keystore n'est pas généré en suivant la documentation (du coup il n'est pas uploadé).

nroselier commented 5 years ago

J'ai testé également sur du Debian 9, en 2.6.3-1 en utilisant le hosts.example.medium.

J'obtiens aussi des certificats uniquement pour ces VMs : ls environments/certs/server/hosts/ localhost medium-access-vm-1 medium-external-vm-1 medium-storage-vm-1

En effet, nous générons des certificats uniquement pour les hosts suivants : hosts-processing hosts-ingest-external hosts-access-external hosts-storage-offer-default hosts-ihm-recette hosts-ihm-demo

Et s'agissant des certificats de timestamping j'obtiens :
find environments/certs/timestamping environments/certs/timestamping environments/certs/timestamping/ca environments/certs/timestamping/ca/ca-intermediate.crt environments/certs/timestamping/ca/ca-root.crt environments/certs/timestamping/vitam environments/certs/timestamping/vitam/secure-logbook.crt environments/certs/timestamping/vitam/secure-logbook.key environments/certs/timestamping/vitam/secure-storage.crt environments/certs/timestamping/vitam/secure-storage.key

Avez-vous la même chose ?

bxaxa commented 5 years ago

Voici ce que j'obtient

ls environments/certs/server/hosts/

localhost  medium-access-vm-1  medium-external-vm-1  medium-storage-vm-1

find environments/certs/timestamping environments/certs/timestamping

environments/certs/timestamping
environments/certs/timestamping/vitam
environments/certs/timestamping/vitam/secure-logbook.crt
environments/certs/timestamping/vitam/secure-storage.crt
environments/certs/timestamping/vitam/secure-storage.key
environments/certs/timestamping/vitam/secure-logbook.key
environments/certs/timestamping/ca
environments/certs/timestamping/ca/ca-root.crt
environments/certs/timestamping/ca/ca-intermediate.crt
environments/certs/timestamping
environments/certs/timestamping/vitam
environments/certs/timestamping/vitam/secure-logbook.crt
environments/certs/timestamping/vitam/secure-storage.crt
environments/certs/timestamping/vitam/secure-storage.key
environments/certs/timestamping/vitam/secure-logbook.key
environments/certs/timestamping/ca
environments/certs/timestamping/ca/ca-root.crt
environments/certs/timestamping/ca/ca-intermediate.crt
croftophile commented 5 years ago

Bonjour, Bug reproduit chez nous avec le fichier d'exemple... Le contournement est facile... Modifier votre inventaire (une fois recopié) et ajouter la machine que vous voulez dans le groupe ansible hosts-ihm-recette (ne contient pas de hosts, théoriquement). A l'issue, relancez le generate_stores.s, qui cette fois doit se terminer comme suit :

[INFO] [generate_stores.sh: main] -------------------------------------------
[INFO] [generate_stores.sh: main] Fin de la génération des stores
mamachine@moninstancedocker:/code/deployment$ echo $?
0

Vous pourrez alors relancer le déploiement.

bxaxa commented 5 years ago

Parfait, ça fonctionne, je laisse l'issue ouverte jusqu'à ce que ce que la correction soit intégrée dans la branche.

vitam-prg commented 4 years ago

A priori corrigé. Je ferme, quitte à réouvrir si besoin