pl853 commented 5 years ago

Qubes OS version:

R4.0.1-RC1

Affected component(s):

Qubes Backup manager

Steps to reproduce the behavior:

Select restore qube from backup in the QubesManager
Select Qube in which backup is found
Select path to backup file
Check verify backup integrity 6 Check encrypted backup box
Fill in decryption password

Expected behavior:

Restore selected backup

Actual behavior:

Display's message: failed to decrypt var/tmp/{the file name}/qubes.xml.000.enc: b'scrypt: Passphrase is incorrect\n'

General notes:

I've used R4.0.1-RC1 for a few day's now and i have not encountered a problem up till now. I tried every option that's available:

Backed up all the Qubes at the same time. (On 5 different qubes and an USB, which was encrypted at first, which didn't work. afterwards i removed the encryption and it still couldn't restore from the usb)
Tried a couple of different encryption passwords for the backup. Which included simple letters and numbers such as 1 or a etc.
Tried checking all the boxes at restore options at the same time and separate.
I also tried an option mentioned in another issue #3211 which had a similar problem: At line 134 in file /usr/lib/python3.5/site-packages/qubesadmin/backup/core3.py: vm.label = vm.properties.pop('label') try: vm.label = self.labels[vm.label] except KeyError: pass

This also didn't work.

Afterwards i re-installed Qubes 4.0 and retried the steps and it worked the first try. So the problem is in R4.0.1-RC1. I also tried to re-install R4.0.1-RC1 and tried it again but with no luck.

Thanks in advance for the help!

Related issues:

3321

3219

#

marmarek commented 5 years ago

I've reproduced the problem already, looking for solution.

pl853 commented 5 years ago

Thankyou, if i can help in any way, ill be there.

On Sat, Nov 10, 2018, 5:22 PM Marek Marczykowski-Górecki < notifications@github.com> wrote:

I've reproduced the problem already, looking for solution.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/4493#issuecomment-437595542, or mute the thread https://github.com/notifications/unsubscribe-auth/AXUmK_oST329LIHtWKsgzkXKZv_W5QPNks5utv0qgaJpZM4YU3nd .

marmarek commented 5 years ago

Does it happen also with backups created on earlier release (4.0, not 4.0.1-rc1)?

pl853 commented 5 years ago

I didnt try that. Ill try it when i get home and get back to you immediatly.

On Sat, Nov 10, 2018, 5:25 PM Marek Marczykowski-Górecki < notifications@github.com> wrote:

Does it happen also with backups created on earlier release (4.0, not 4.0.1-rc1)?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/4493#issuecomment-437595833, or mute the thread https://github.com/notifications/unsubscribe-auth/AXUmK9pBmpx2Kmt7MS9A8WeoPdShMA2Kks5utv36gaJpZM4YU3nd .

marmarek commented 5 years ago

No need, it's the backup code broken, not the restore one. Specifically, this change broke it: https://github.com/QubesOS/qubes-core-admin/commit/4e49b951ceeaa6b081062f4acdea7e7f2c771300#diff-d5cd0937e32eff778591ab56cd19526eR260

pl853 commented 5 years ago

Okay, should i try to reverse it or will it compremise security?

On Sat, Nov 10, 2018, 5:48 PM Marek Marczykowski-Górecki < notifications@github.com> wrote:

No need, it's the backup code broken, not the restore one. Specifically, this change broke it: QubesOS/qubes-core-admin@4e49b95

diff-d5cd0937e32eff778591ab56cd19526eR260

https://github.com/QubesOS/qubes-core-admin/commit/4e49b951ceeaa6b081062f4acdea7e7f2c771300#diff-d5cd0937e32eff778591ab56cd19526eR260

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/4493#issuecomment-437597549, or mute the thread https://github.com/notifications/unsubscribe-auth/AXUmK-AwvYOLrFn10bTkUWPjaLAYzpBxks5utwNGgaJpZM4YU3nd .

qubesos-bot commented 5 years ago

Automated announcement from builder-github

The package qubes-core-dom0-4.0.35-1.fc25 has been pushed to the r4.0 testing repository for dom0. To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

qubesos-bot commented 5 years ago

Automated announcement from builder-github

The package qubes-core-dom0-4.0.37-1.fc25 has been pushed to the r4.0 stable repository for dom0. To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

andrewdavidwong commented 1 year ago

Reopening due to apparent regression reported in #8083. I also just noticed that the same error message was reported in #6227. Updating title to be more descriptive.

akkuladezeit commented 1 year ago

Any News on This? In 4.1.2 its still present and Backups Not Possible. Are Backup Really that unimportant?

andrewdavidwong commented 1 year ago

Any News on This? In 4.1.2 its still present and Backups Not Possible. Are Backup Really that unimportant?

Of course backups are important (hence the high priority label on this issue). However, this issue seems to be hardware-specific (based on the comments on #6227 and the fact that backups still appear to be working on most systems), so it's not accurate to say that backups are "not possible." Perhaps that also makes the problem more difficult for the developers to diagnose and fix.

akkuladezeit commented 1 year ago

Ok i can give some Info About my Hardware.

I got an AMD Ryzen 5 (1600X) with 32gb RAM The Main Drive is an NVME with ~3000 Mb/s Speed Second Drive is a 10 Tb WD Red

Backup is on Both Backup-Destinations.

The CLI Version also fails and the -v dont give much hints.

Is there a way to get More Infos out of the process ?

joetretter commented 1 year ago

I have played around a little bit with that, because I have the same problem on my workstation. I notice that I also have an AMD processor (AMD Ryzen 7 1700 Eight-Core Processor) on the machine where I experience the restore issues.

I followed the "emergency backup recovery" document. (https://www.qubes-os.org/doc/backup-emergency-restore-v4/) and it seems that there is an issue that vanishes when runing one of the steps a second time without changing any of the inputs.

Here what I observed - this is just an un-edited copy/paste from my terminal (Yes, my password was abc):

....
user@admin:~/bky/wrk$ find vm3 -name 'private.img.*.enc' | sort -V | while read f_enc; do     f_dec=${f_enc%.enc};     echo "$backup_id!$f_dec!$backup_pass" | scrypt dec -P $f_enc || break;     done | gzip -d | tar -xv vm3/private.img
scrypt: Passphrase is incorrect

gzip: stdin: unexpected end of file
tar: This does not look like a tar archive
tar: vm3/private.img: Not found in archive
tar: Exiting with failure status due to previous errors
user@admin:~/bky/wrk$ echo $backup_pass
abc
user@admin:~/bky/wrk$ find vm3 -name 'private.img.*.enc' | sort -V | while read f_enc; do     f_dec=${f_enc%.enc};     echo "$backup_id!$f_dec!$backup_pass" | scrypt dec -P $f_enc || break;     done | gzip -d | tar -xv vm3/private.img
vm3/private.img
user@admin:~/bky/wrk$ sudo mount -o loop vm3/private.img /mnt/img/
user@admin:~/bky/wrk$ find /mnt/img/home/user/
/mnt/img/home/user/
/mnt/img/home/user/Templates
/mnt/img/home/user/.bashrc
....

rustybird commented 1 year ago

@joetretter:

it seems that there is an issue that vanishes when runing one of the steps a second time without changing any of the inputs.

That smells like a hardware error, e.g. faulty RAM and/or overheating.

akkuladezeit commented 9 months ago

Is there any way to Backup without using the Tool in Qubes3 it was easy but on 4.1 it seams impossible? I just wont to migrate to a New Hardware but every Restore fails ... and i dont think it will be fixed in near Future

akkuladezeit commented 9 months ago

@joetretter:

it seems that there is an issue that vanishes when runing one of the steps a second time without changing any of the inputs.

That smells like a hardware error, e.g. faulty RAM and/or overheating.

I dont observed any System Crash.. so i dont thinks That it is a ram problem.

Its more likely a Problem of amd ryzen series 1 ?

joetretter commented 9 months ago

I have been able to drill down this problem to scrypt not working properly on my AMD Ryzen 7 1700 Eight-Core Processor. Even downloading the code and running the tests, I could see that it fails on this computer - but it didn't always fail at the same test. To be able to move my VMS to another computer, I worked around this issue by temporarily replacing the scrypt executable on the source and target machine with this script:

#!/bin/bash
# 2023-04-30 Joe Tretter, fake replacement for scrypt to work around backup restore issue in qubes-os
# THIS IS UNSAFE BECAUSE IT DOES NOT ENCRYPT!!!
if [[ "$1" == "enc" ]]
then 
>&2 echo -en "Please enter passphrase: "
>&2 echo -en "Please confirm passphrase: "
else
>&2 echo -en "Please enter passphrase: "
fi
lastArgNo=$#
secondLastArgNo=$((lastArgNo-1))
in=${!secondLastArgNo}
out=${!lastArgNo}
if [[ "-" = "${in}" ]]; then 
cat > $out
else
cat $in > $out
fi

rustybird commented 9 months ago

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

joetretter commented 9 months ago

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.

DemiMarie commented 9 months ago

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.

That smells like either:

Memory corruption.
Xen mishandling some instruction.

joetretter commented 9 months ago

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.

That smells like either:

Memory corruption.

Xen mishandling some instruction.

Memory corruption is unlikely; the machine has 2x16GB, and I have reproduced the problem with every permutation of 1- and 2- RAM sticks across both RAM slots on the mainboard. I have also repeatedly executed the latest memtest86+. My specific machine is a Dell Inspiron 5675.

DemiMarie commented 9 months ago

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.

That smells like either:

Memory corruption.

Xen mishandling some instruction.

Memory corruption is unlikely; the machine has 2x16GB, and I have reproduced the problem with every permutation of 1- and 2- RAM sticks across both RAM slots on the mainboard. I have also repeatedly executed the latest memtest86+. My specific machine is a Dell Inspiron 5675.

Can you reproduce this problem without Xen using the Qubes-provided scrypt and Linux kernel? Qubes should boot without Xen, but no VMs will start.

joetretter commented 8 months ago

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.

That smells like either:

Memory corruption.

Xen mishandling some instruction.

Memory corruption is unlikely; the machine has 2x16GB, and I have reproduced the problem with every permutation of 1- and 2- RAM sticks across both RAM slots on the mainboard. I have also repeatedly executed the latest memtest86+. My specific machine is a Dell Inspiron 5675.

Can you reproduce this problem without Xen using the Qubes-provided scrypt and Linux kernel? Qubes should boot without Xen, but no VMs will start.

I don't have Qubes installed on that computer at the moment, I can try that out when I have some time... What do I need to do to boot without Xen?

rustybird commented 8 months ago

What do I need to do to boot without Xen?

https://github.com/QubesOS/qubes-issues/issues/8574#issuecomment-1751814746

akkuladezeit commented 8 months ago

I have been able to drill down this problem to scrypt not working properly on my AMD Ryzen 7 1700 Eight-Core Processor.

Even downloading the code and running the tests, I could see that it fails on this computer - but it didn't always fail at the same test. To be able to move my VMS to another computer, I worked around this issue by temporarily replacing the scrypt executable on the source and target machine with this script:
#!/bin/bash

# 2023-04-30 Joe Tretter, fake replacement for scrypt to work around backup restore issue in qubes-os

# THIS IS UNSAFE BECAUSE IT DOES NOT ENCRYPT!!!

if [[ "$1" == "enc" ]]

then 

>&2 echo -en "Please enter passphrase: "

>&2 echo -en "Please confirm passphrase: "

else

>&2 echo -en "Please enter passphrase: "

fi

lastArgNo=$#

secondLastArgNo=$((lastArgNo-1))

in=${!secondLastArgNo}

out=${!lastArgNo}

if [[ "-" = "${in}" ]]; then 

cat > $out

else

cat $in > $out

fi

Can you give some instruction how to repace it ? That would be nice 🥹

Maybe an Backup Option without any encrypt would be nice for such Problems..

joetretter commented 8 months ago

Can you give some instruction how to repace it ? That would be nice 🥹
1) Open a terminal in your personal vm
2) run command 
cat > scrypt
3) paste the script and push Ctrl+D
4) verify the integrity with md5sum scrypt (the hash is supposed to be 1b3ca107905f023a65960ce0829a0287)
5) open a dom0 terminal
6) enter the following commands:
sudo su
cd /usr/bin
mv scrypt scrypt.original
qvm-run --pass-io personal 'cat /home/user/scrypt' > scrypt
chmod 777 scrypt
Maybe an Backup Option without any encrypt would be nice for such Problems.. Yes, I agree - funny enough the restore has an unencrypted option even though there is no such option in the backup ;)

joetretter commented 8 months ago

Can you reproduce this problem without Xen using the Qubes-provided scrypt and Linux kernel? Qubes should boot without Xen, but no VMs will start.

I have tried it. Without Xen the issue does not happen (see screenshots).

NoXen WithXen

DemiMarie commented 8 months ago

@joetretter I think you found a genuine bug in Xen, then. Please report this to xen-devel@lists.xenproject.org.

akkuladezeit commented 8 months ago

Can you give some instruction how to repace it ? That would be nice 🥹
1) Open a terminal in your personal vm

2) run command 

cat > scrypt

3) paste the script and push Ctrl+D

4) verify the integrity with md5sum scrypt (the hash is supposed to be 1b3ca107905f023a65960ce0829a0287)

5) open a dom0 terminal

6) enter the following commands:

sudo su

cd /usr/bin

mv scrypt scrypt.original

qvm-run --pass-io personal 'cat /home/user/scrypt' > scrypt

chmod 777 scrypt
Maybe an Backup Option without any encrypt would be nice for such Problems..

Yes, I agree - funny enough the restore has an unencrypted option even though there is no such option in the backup ;)

Thanks Works Great

Eric678 commented 8 months ago

From above closed issue that included 4.2 in affected: add Ryzen 9 to affected hardware - this does seem to be AMD specific, so is different from the original issue in 2018.

The very odd thing is, I know I tested this in 4.2.0-rc1 and there was no problem. I was having problems writing USB storage and so did quite a bit of testing. Going back, I have a full backup from early October that is ~80GB compressed that verifies OK, while one a week later after doing an upgrade fails with "scrypt:Input is not valid scrypt-encrypted block" maybe half way through. Not sure what versions were then. Now a backup of a fresh 4.2.0 instal reliably fails quickly, so it looks like quite a recent regression. Add that to the sudden appearance of app qube kernel panics in 6.1.62 and stable 4.2.0 is looking bad.

DemiMarie commented 8 months ago

@joetretter Does the test suite pass in a Qubes VM? If so, the problem is specific to Xen paravirtualization.

joetretter commented 8 months ago

@joetretter Does the test suite pass in a Qubes VM? If so, the problem is specific to Xen paravirtualization.

The test is not passing in a Qubes-VM either. I am in contact with the XEN folks and they asked me to try with the latest microcode. I have spent about three days and I'm afraid I am not able to get the latest microcode working in Qubes, I can't seem to figure out the right way to make it known to the kernel. As it's been released in parallel to my testing, I have upgraded to 4.2 in the hope that the latest firmware would be included or the problem be solved. Unfortunately it isn't. Can the Qubes project get the latest AMD firmware into 4.2, or can you point me to a resource how I can get it working? I am running Manjaro (archlinux) on the same machine and there I have the latest firmware (see attachments) UcodeVersion-Qubes4 2 UcodeVersion-Manjaro

marmarek commented 8 months ago

@joetretter do you have most recent dracut package installed (version 059-5) in dom0?

joetretter commented 8 months ago

@joetretter do you have most recent dracut package installed (version 059-5) in dom0?

Yes, I have dracut 059-5.fc37

DemiMarie commented 8 months ago

@joetretter You’ll need to get a firmware update from your hardware vendor. R4.2 has support for applying such updates, so long as the vendor makes them available via fwupd.

@marmarek I’m starting to wonder if we should offer late loading of firmware on AMD systems too, so long as we can find the firmware somewhere. AMD doesn’t test it on most client parts so it isn’t great, but on systems like what @joetretter has it might be the difference between being able to use Qubes OS and not being able to.

brendanhoar commented 8 months ago

@joetretter do you have most recent dracut package installed (version 059-5) in dom0?

I just wanted to note that on my intel Thinkpad P52 with the release version of R4.2 and kernel latest, I also have that most recent version of dracut.

However, no microcode is being loaded (according to xen dmesg and Linux dmesg). I've even performed a sudo dracut --regenerate-all --force and rebooted with no change.

B

marmarek commented 8 months ago

However, no microcode is being loaded (according to xen dmesg and Linux dmesg). I've even performed a sudo dracut --regenerate-all --force and rebooted with no change.

It gets loaded only if there is a newer version available. Most likely your firmware already loads latest version (as it should!) so OS doesn't need to load any updates.

QubesOS / qubes-issues

Backup restore finishes with error: `failed to decrypt [...]: b'scrypt: Passphrase is incorrect` #4493

Qubes OS version:

Affected component(s):

Steps to reproduce the behavior:

Expected behavior:

Actual behavior:

General notes:

Related issues:

3321

3219

diff-d5cd0937e32eff778591ab56cd19526eR260