Closed termie closed 13 years ago
termie
commented
13 years ago (by berendt) seems to be a problem with /var/lib/nova/CA. I copied all files from an other system and now I can create decrypted images. Can anybody give me a hint what we have done in the wrong way when moving nova-objectstore to an other system?
termie
commented
13 years ago (by ttx) No clue, but can't reproduce it with current trunk, looks like a local issue ?
termie
commented
13 years ago (by berendt) I think the problem is that there are certificates registered in the table certificates. I moved the objectstore to an other systems and so he couldn't find the those certificates on the new system. After copien those certificates to the new systems all is working fine.
mysql> select * from certificates; +---------------------+------------+------------+---------+----+---------+------------+-----------------------------------+ | created_at | updated_at | deleted_at | deleted | id | user_id | project_id | file_name | +---------------------+------------+------------+---------+----+---------+------------+-----------------------------------+ | 2011-02-24 08:21:01 | NULL | NULL | 0 | 1 | berendt | testing | /var/lib/nova/CA/newcerts/12 .pem | | 2011-02-24 08:37:47 | NULL | NULL | 0 | 2 | berendt | openstack | /var/lib/nova/CA/newcerts/13 .pem | | 2011-02-24 08:38:01 | NULL | NULL | 0 | 3 | berendt | openstack | /var/lib/nova/CA/newcerts/14 .pem | +---------------------+------------+------------+---------+----+---------+------------+-----------------------------------+
termie
commented
13 years ago (by vishvananda) you have to run api and obectstore on the same host currently.
Vish On Feb 25, 2011, at 5:35 AM, Christian Berendt wrote:
I think the problem is that there are certificates registered in the table certificates. I moved the objectstore to an other systems and so he couldn't find the those certificates on the new system. After copien those certificates to the new systems all is working fine.
mysql> select * from certificates; +---------------------+------------+------------+---------+----+---------+------------+-----------------------------------+ | created_at | updated_at | deleted_at | deleted | id | user_id | project_id | file_name | +---------------------+------------+------------+---------+----+---------+------------+-----------------------------------+ | 2011-02-24 08:21:01 | NULL | NULL | 0 | 1 | berendt | testing | /var/lib/nova/CA/newcerts/12 .pem | | 2011-02-24 08:37:47 | NULL | NULL | 0 | 2 | berendt | openstack | /var/lib/nova/CA/newcerts/13 .pem | | 2011-02-24 08:38:01 | NULL | NULL | 0 | 3 | berendt | openstack | /var/lib/nova/CA/newcerts/14 .pem | +---------------------+------------+------------+---------+----+---------+------------+-----------------------------------+
\ Changed in: nova Status: Incomplete => Invalid
You received this bug notification because you are a member of Nova Bug Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/724853
Title: after uploading images to nova-objectstore they are only decrypting
Status in OpenStack Compute (Nova): Invalid
Bug description: In the past new images were available after a few seconds or minutes after finished the upload. Now we only see the state "decrypting" and only files "image.encrypted". We're using bzrrev 732. Also we don't have running processes on the nova-objectstore system with high cpu usage.
In objectstore.log we have the following error:
---snip--- 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] Process Process-3: 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] Traceback (most recent call last): 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/multiprocessing/process.py", line 231, in _bootstrap 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] self.run() 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] self._target(_self._args, *_self._kwargs) 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/site-packages/nova/objectstore/image.py", line 242, in register_aws_image 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] cloud_private_key, decrypted_filename) 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/site-packages/nova/objectstore/image.py", line 263, in decrypt_image 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] % err) 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] Error: Failed to decrypt private key: RSA operation error 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] 1918:error:0407106B:rsa routines:RSA_padding_check_PKCS1_type_2:block type is not 02:rsa_pk1.c:190: 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] 1918:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:rsa_eay.c:596: ---snap---
I think the CA stuff looks good: ---snip--- chronos:/var/lib/nova/CA # ./genrootca.sh Not installing, it's already done. ---snap---
---snip--- IMAGE ami-0h5aejwo debian_kernel/vmlinuz-2.6.28-11-generic.manifest.xml testing decrypting private x86_64 kernel true IMAGE ami-0zt2o6tu debian_image/debian.5-0.x86-64.img.manifest.xml testing decrypting private x86_64 machine ami-qbqcxp5d ami-0h5aejwo IMAGE ami-qbqcxp5d debian_ramdisk/initrd.img-2.6.28-11-generic.manifest.xml testing decrypting private x86_64 ramdisk true
---snap------snip--- chronos:/srv/images # find . . ./ami-0h5aejwo ./ami-0h5aejwo/image.encrypted ./ami-0h5aejwo/info.json ./ami-0zt2o6tu ./ami-0zt2o6tu/image.encrypted ./ami-0zt2o6tu/info.json ./lost+found ./ami-qbqcxp5d ./ami-qbqcxp5d/image.encrypted ./ami-qbqcxp5d/info.json ---snap---
termie
commented
13 years ago (by berendt) Yes, I figured that out after some coffee ;) But the problem is that I moved nova-objectstore, nova-api and nova-scheduler to an other host. When moving all that services you have to also move /var/lib/nova/CA to the new system.
termie
commented
13 years ago (by vishvananda) ah, yes, that is correct.
Vish
On Feb 25, 2011, at 11:38 AM, Christian Berendt wrote:
Yes, I figured that out after some coffee ;) But the problem is that I moved nova-objectstore, nova-api and nova-scheduler to an other host. When moving all that services you have to also move /var/lib/nova/CA to the new system.
You received this bug notification because you are a member of Nova Bug Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/724853
Title: after uploading images to nova-objectstore they are only decrypting
Status in OpenStack Compute (Nova): Invalid
Bug description: In the past new images were available after a few seconds or minutes after finished the upload. Now we only see the state "decrypting" and only files "image.encrypted". We're using bzrrev 732. Also we don't have running processes on the nova-objectstore system with high cpu usage.
In objectstore.log we have the following error:
---snip--- 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] Process Process-3: 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] Traceback (most recent call last): 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/multiprocessing/process.py", line 231, in _bootstrap 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] self.run() 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] self._target(_self._args, *_self._kwargs) 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/site-packages/nova/objectstore/image.py", line 242, in register_aws_image 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] cloud_private_key, decrypted_filename) 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/site-packages/nova/objectstore/image.py", line 263, in decrypt_image 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] % err) 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] Error: Failed to decrypt private key: RSA operation error 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] 1918:error:0407106B:rsa routines:RSA_padding_check_PKCS1_type_2:block type is not 02:rsa_pk1.c:190: 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] 1918:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:rsa_eay.c:596: ---snap---
I think the CA stuff looks good: ---snip--- chronos:/var/lib/nova/CA # ./genrootca.sh Not installing, it's already done. ---snap---
---snip--- IMAGE ami-0h5aejwo debian_kernel/vmlinuz-2.6.28-11-generic.manifest.xml testing decrypting private x86_64 kernel true IMAGE ami-0zt2o6tu debian_image/debian.5-0.x86-64.img.manifest.xml testing decrypting private x86_64 machine ami-qbqcxp5d ami-0h5aejwo IMAGE ami-qbqcxp5d debian_ramdisk/initrd.img-2.6.28-11-generic.manifest.xml testing decrypting private x86_64 ramdisk true
---snap------snip--- chronos:/srv/images # find . . ./ami-0h5aejwo ./ami-0h5aejwo/image.encrypted ./ami-0h5aejwo/info.json ./ami-0zt2o6tu ./ami-0zt2o6tu/image.encrypted ./ami-0zt2o6tu/info.json ./lost+found ./ami-qbqcxp5d ./ami-qbqcxp5d/image.encrypted ./ami-qbqcxp5d/info.json ---snap---
In the past new images were available after a few seconds or minutes after finished the upload. Now we only see the state "decrypting" and only files "image.encrypted". We're using bzrrev 732. Also we don't have running processes on the nova-objectstore system with high cpu usage.
In objectstore.log we have the following error:
---snip--- 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] Process Process-3: 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] Traceback (most recent call last): 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/multiprocessing/process.py", line 231, in _bootstrap 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] self.run() 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] self._target(_self._args, *_self._kwargs) 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/site-packages/nova/objectstore/image.py", line 242, in register_aws_image 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] cloud_private_key, decrypted_filename) 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] File "/usr/lib64/python2.6/site-packages/nova/objectstore/image.py", line 263, in decrypt_image 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] % err) 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] Error: Failed to decrypt private key: RSA operation error 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] 1918:error:0407106B:rsa routines:RSA_padding_check_PKCS1_type_2:block type is not 02:rsa_pk1.c:190: 2011-02-25 10:46:16+0100 [HTTPChannel,4,192.168.2.130] 1918:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:rsa_eay.c:596: ---snap---
I think the CA stuff looks good: ---snip--- chronos:/var/lib/nova/CA # ./genrootca.sh Not installing, it's already done. ---snap---
---snip--- IMAGE ami-0h5aejwo debian_kernel/vmlinuz-2.6.28-11-generic.manifest.xml testing decrypting private x86_64 kernel true IMAGE ami-0zt2o6tu debian_image/debian.5-0.x86-64.img.manifest.xml testing decrypting private x86_64 machine ami-qbqcxp5d ami-0h5aejwo IMAGE ami-qbqcxp5d debian_ramdisk/initrd.img-2.6.28-11-generic.manifest.xml testing decrypting private x86_64 ramdisk true
---snap---
---snip--- chronos:/srv/images # find . . ./ami-0h5aejwo ./ami-0h5aejwo/image.encrypted ./ami-0h5aejwo/info.json ./ami-0zt2o6tu ./ami-0zt2o6tu/image.encrypted ./ami-0zt2o6tu/info.json ./lost+found ./ami-qbqcxp5d ./ami-qbqcxp5d/image.encrypted ./ami-qbqcxp5d/info.json ---snap---
Imported from Launchpad using lp2gh.