Open pl853 opened 5 years ago
I've reproduced the problem already, looking for solution.
Thankyou, if i can help in any way, ill be there.
On Sat, Nov 10, 2018, 5:22 PM Marek Marczykowski-Górecki < notifications@github.com> wrote:
I've reproduced the problem already, looking for solution.
— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/4493#issuecomment-437595542, or mute the thread https://github.com/notifications/unsubscribe-auth/AXUmK_oST329LIHtWKsgzkXKZv_W5QPNks5utv0qgaJpZM4YU3nd .
Does it happen also with backups created on earlier release (4.0, not 4.0.1-rc1)?
I didnt try that. Ill try it when i get home and get back to you immediatly.
On Sat, Nov 10, 2018, 5:25 PM Marek Marczykowski-Górecki < notifications@github.com> wrote:
Does it happen also with backups created on earlier release (4.0, not 4.0.1-rc1)?
— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/4493#issuecomment-437595833, or mute the thread https://github.com/notifications/unsubscribe-auth/AXUmK9pBmpx2Kmt7MS9A8WeoPdShMA2Kks5utv36gaJpZM4YU3nd .
No need, it's the backup code broken, not the restore one. Specifically, this change broke it: https://github.com/QubesOS/qubes-core-admin/commit/4e49b951ceeaa6b081062f4acdea7e7f2c771300#diff-d5cd0937e32eff778591ab56cd19526eR260
Okay, should i try to reverse it or will it compremise security?
On Sat, Nov 10, 2018, 5:48 PM Marek Marczykowski-Górecki < notifications@github.com> wrote:
No need, it's the backup code broken, not the restore one. Specifically, this change broke it: QubesOS/qubes-core-admin@4e49b95
diff-d5cd0937e32eff778591ab56cd19526eR260
— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/QubesOS/qubes-issues/issues/4493#issuecomment-437597549, or mute the thread https://github.com/notifications/unsubscribe-auth/AXUmK-AwvYOLrFn10bTkUWPjaLAYzpBxks5utwNGgaJpZM4YU3nd .
Automated announcement from builder-github
The package qubes-core-dom0-4.0.35-1.fc25
has been pushed to the r4.0
testing repository for dom0.
To test this update, please install it with the following command:
sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing
Automated announcement from builder-github
The package qubes-core-dom0-4.0.37-1.fc25
has been pushed to the r4.0
stable repository for dom0.
To install this update, please use the standard update command:
sudo qubes-dom0-update
Or update dom0 via Qubes Manager.
Reopening due to apparent regression reported in #8083. I also just noticed that the same error message was reported in #6227. Updating title to be more descriptive.
Any News on This? In 4.1.2 its still present and Backups Not Possible. Are Backup Really that unimportant?
Any News on This? In 4.1.2 its still present and Backups Not Possible. Are Backup Really that unimportant?
Of course backups are important (hence the high priority label on this issue). However, this issue seems to be hardware-specific (based on the comments on #6227 and the fact that backups still appear to be working on most systems), so it's not accurate to say that backups are "not possible." Perhaps that also makes the problem more difficult for the developers to diagnose and fix.
Ok i can give some Info About my Hardware.
I got an AMD Ryzen 5 (1600X) with 32gb RAM The Main Drive is an NVME with ~3000 Mb/s Speed Second Drive is a 10 Tb WD Red
Backup is on Both Backup-Destinations.
The CLI Version also fails and the -v dont give much hints.
Is there a way to get More Infos out of the process ?
I have played around a little bit with that, because I have the same problem on my workstation. I notice that I also have an AMD processor (AMD Ryzen 7 1700 Eight-Core Processor) on the machine where I experience the restore issues.
I followed the "emergency backup recovery" document. (https://www.qubes-os.org/doc/backup-emergency-restore-v4/) and it seems that there is an issue that vanishes when runing one of the steps a second time without changing any of the inputs.
Here what I observed - this is just an un-edited copy/paste from my terminal (Yes, my password was abc):
....
user@admin:~/bky/wrk$ find vm3 -name 'private.img.*.enc' | sort -V | while read f_enc; do f_dec=${f_enc%.enc}; echo "$backup_id!$f_dec!$backup_pass" | scrypt dec -P $f_enc || break; done | gzip -d | tar -xv vm3/private.img
scrypt: Passphrase is incorrect
gzip: stdin: unexpected end of file
tar: This does not look like a tar archive
tar: vm3/private.img: Not found in archive
tar: Exiting with failure status due to previous errors
user@admin:~/bky/wrk$ echo $backup_pass
abc
user@admin:~/bky/wrk$ find vm3 -name 'private.img.*.enc' | sort -V | while read f_enc; do f_dec=${f_enc%.enc}; echo "$backup_id!$f_dec!$backup_pass" | scrypt dec -P $f_enc || break; done | gzip -d | tar -xv vm3/private.img
vm3/private.img
user@admin:~/bky/wrk$ sudo mount -o loop vm3/private.img /mnt/img/
user@admin:~/bky/wrk$ find /mnt/img/home/user/
/mnt/img/home/user/
/mnt/img/home/user/Templates
/mnt/img/home/user/.bashrc
....
@joetretter:
it seems that there is an issue that vanishes when runing one of the steps a second time without changing any of the inputs.
That smells like a hardware error, e.g. faulty RAM and/or overheating.
Is there any way to Backup without using the Tool in Qubes3 it was easy but on 4.1 it seams impossible? I just wont to migrate to a New Hardware but every Restore fails ... and i dont think it will be fixed in near Future
@joetretter:
it seems that there is an issue that vanishes when runing one of the steps a second time without changing any of the inputs.
That smells like a hardware error, e.g. faulty RAM and/or overheating.
I dont observed any System Crash.. so i dont thinks That it is a ram problem.
Its more likely a Problem of amd ryzen series 1 ?
I have been able to drill down this problem to scrypt not working properly on my AMD Ryzen 7 1700 Eight-Core Processor. Even downloading the code and running the tests, I could see that it fails on this computer - but it didn't always fail at the same test. To be able to move my VMS to another computer, I worked around this issue by temporarily replacing the scrypt executable on the source and target machine with this script:
#!/bin/bash
# 2023-04-30 Joe Tretter, fake replacement for scrypt to work around backup restore issue in qubes-os
# THIS IS UNSAFE BECAUSE IT DOES NOT ENCRYPT!!!
if [[ "$1" == "enc" ]]
then
>&2 echo -en "Please enter passphrase: "
>&2 echo -en "Please confirm passphrase: "
else
>&2 echo -en "Please enter passphrase: "
fi
lastArgNo=$#
secondLastArgNo=$((lastArgNo-1))
in=${!secondLastArgNo}
out=${!lastArgNo}
if [[ "-" = "${in}" ]]; then
cat > $out
else
cat $in > $out
fi
Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?
Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?
Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.
Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?
Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.
That smells like either:
Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?
Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.
That smells like either:
- Memory corruption.
- Xen mishandling some instruction.
Memory corruption is unlikely; the machine has 2x16GB, and I have reproduced the problem with every permutation of 1- and 2- RAM sticks across both RAM slots on the mainboard. I have also repeatedly executed the latest memtest86+. My specific machine is a Dell Inspiron 5675.
Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?
Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.
That smells like either:
- Memory corruption.
- Xen mishandling some instruction.
Memory corruption is unlikely; the machine has 2x16GB, and I have reproduced the problem with every permutation of 1- and 2- RAM sticks across both RAM slots on the mainboard. I have also repeatedly executed the latest memtest86+. My specific machine is a Dell Inspiron 5675.
Can you reproduce this problem without Xen using the Qubes-provided scrypt and Linux kernel? Qubes should boot without Xen, but no VMs will start.
Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?
Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.
That smells like either:
- Memory corruption.
- Xen mishandling some instruction.
Memory corruption is unlikely; the machine has 2x16GB, and I have reproduced the problem with every permutation of 1- and 2- RAM sticks across both RAM slots on the mainboard. I have also repeatedly executed the latest memtest86+. My specific machine is a Dell Inspiron 5675.
Can you reproduce this problem without Xen using the Qubes-provided scrypt and Linux kernel? Qubes should boot without Xen, but no VMs will start.
I don't have Qubes installed on that computer at the moment, I can try that out when I have some time... What do I need to do to boot without Xen?
What do I need to do to boot without Xen?
https://github.com/QubesOS/qubes-issues/issues/8574#issuecomment-1751814746
I have been able to drill down this problem to scrypt not working properly on my AMD Ryzen 7 1700 Eight-Core Processor.
Even downloading the code and running the tests, I could see that it fails on this computer - but it didn't always fail at the same test. To be able to move my VMS to another computer, I worked around this issue by temporarily replacing the scrypt executable on the source and target machine with this script:
#!/bin/bash # 2023-04-30 Joe Tretter, fake replacement for scrypt to work around backup restore issue in qubes-os # THIS IS UNSAFE BECAUSE IT DOES NOT ENCRYPT!!! if [[ "$1" == "enc" ]] then >&2 echo -en "Please enter passphrase: " >&2 echo -en "Please confirm passphrase: " else >&2 echo -en "Please enter passphrase: " fi lastArgNo=$# secondLastArgNo=$((lastArgNo-1)) in=${!secondLastArgNo} out=${!lastArgNo} if [[ "-" = "${in}" ]]; then cat > $out else cat $in > $out fi
Can you give some instruction how to repace it ? That would be nice 🥹
Maybe an Backup Option without any encrypt would be nice for such Problems..
Can you give some instruction how to repace it ? That would be nice 🥹
1) Open a terminal in your personal vm 2) run command cat > scrypt 3) paste the script and push Ctrl+D 4) verify the integrity with md5sum scrypt (the hash is supposed to be 1b3ca107905f023a65960ce0829a0287) 5) open a dom0 terminal 6) enter the following commands: sudo su cd /usr/bin mv scrypt scrypt.original qvm-run --pass-io personal 'cat /home/user/scrypt' > scrypt chmod 777 scrypt
Maybe an Backup Option without any encrypt would be nice for such Problems.. Yes, I agree - funny enough the restore has an unencrypted option even though there is no such option in the backup ;)
Can you reproduce this problem without Xen using the Qubes-provided scrypt and Linux kernel? Qubes should boot without Xen, but no VMs will start.
I have tried it. Without Xen the issue does not happen (see screenshots).
@joetretter I think you found a genuine bug in Xen, then. Please report this to xen-devel@lists.xenproject.org.
Can you give some instruction how to repace it ? That would be nice 🥹
1) Open a terminal in your personal vm 2) run command cat > scrypt 3) paste the script and push Ctrl+D 4) verify the integrity with md5sum scrypt (the hash is supposed to be 1b3ca107905f023a65960ce0829a0287) 5) open a dom0 terminal 6) enter the following commands: sudo su cd /usr/bin mv scrypt scrypt.original qvm-run --pass-io personal 'cat /home/user/scrypt' > scrypt chmod 777 scrypt
Maybe an Backup Option without any encrypt would be nice for such Problems..
Yes, I agree - funny enough the restore has an unencrypted option even though there is no such option in the backup ;)
Thanks Works Great
From above closed issue that included 4.2 in affected: add Ryzen 9 to affected hardware - this does seem to be AMD specific, so is different from the original issue in 2018.
The very odd thing is, I know I tested this in 4.2.0-rc1 and there was no problem. I was having problems writing USB storage and so did quite a bit of testing. Going back, I have a full backup from early October that is ~80GB compressed that verifies OK, while one a week later after doing an upgrade fails with "scrypt:Input is not valid scrypt-encrypted block" maybe half way through. Not sure what versions were then. Now a backup of a fresh 4.2.0 instal reliably fails quickly, so it looks like quite a recent regression. Add that to the sudden appearance of app qube kernel panics in 6.1.62 and stable 4.2.0 is looking bad.
@joetretter Does the test suite pass in a Qubes VM? If so, the problem is specific to Xen paravirtualization.
@joetretter Does the test suite pass in a Qubes VM? If so, the problem is specific to Xen paravirtualization.
The test is not passing in a Qubes-VM either. I am in contact with the XEN folks and they asked me to try with the latest microcode. I have spent about three days and I'm afraid I am not able to get the latest microcode working in Qubes, I can't seem to figure out the right way to make it known to the kernel. As it's been released in parallel to my testing, I have upgraded to 4.2 in the hope that the latest firmware would be included or the problem be solved. Unfortunately it isn't. Can the Qubes project get the latest AMD firmware into 4.2, or can you point me to a resource how I can get it working? I am running Manjaro (archlinux) on the same machine and there I have the latest firmware (see attachments)
@joetretter do you have most recent dracut package installed (version 059-5) in dom0?
@joetretter do you have most recent dracut package installed (version 059-5) in dom0?
Yes, I have dracut 059-5.fc37
@joetretter You’ll need to get a firmware update from your hardware vendor. R4.2 has support for applying such updates, so long as the vendor makes them available via fwupd.
@marmarek I’m starting to wonder if we should offer late loading of firmware on AMD systems too, so long as we can find the firmware somewhere. AMD doesn’t test it on most client parts so it isn’t great, but on systems like what @joetretter has it might be the difference between being able to use Qubes OS and not being able to.
@joetretter do you have most recent dracut package installed (version 059-5) in dom0?
I just wanted to note that on my intel Thinkpad P52 with the release version of R4.2 and kernel latest, I also have that most recent version of dracut.
However, no microcode is being loaded (according to xen dmesg and Linux dmesg). I've even performed a sudo dracut --regenerate-all --force and rebooted with no change.
B
However, no microcode is being loaded (according to xen dmesg and Linux dmesg). I've even performed a sudo dracut --regenerate-all --force and rebooted with no change.
It gets loaded only if there is a newer version available. Most likely your firmware already loads latest version (as it should!) so OS doesn't need to load any updates.
Qubes OS version:
R4.0.1-RC1
Affected component(s):
Qubes Backup manager
Steps to reproduce the behavior:
Expected behavior:
Restore selected backup
Actual behavior:
Display's message: failed to decrypt var/tmp/{the file name}/qubes.xml.000.enc: b'scrypt: Passphrase is incorrect\n'
General notes:
I've used R4.0.1-RC1 for a few day's now and i have not encountered a problem up till now. I tried every option that's available:
This also didn't work.
Afterwards i re-installed Qubes 4.0 and retried the steps and it worked the first try. So the problem is in R4.0.1-RC1. I also tried to re-install R4.0.1-RC1 and tried it again but with no luck.
Thanks in advance for the help!
Related issues:
3321
3219
#