search

MichaIng / DietPi

Lightweight justice for your single-board computer!
https://dietpi.com/
GNU General Public License v2.0
4.76k stars 494 forks source link

Images | Update all images #2026

Closed Fourdee closed 5 years ago

Fourdee commented 6 years ago

As per: https://github.com/Fourdee/DietPi/issues/2020

Some users are experiencing this during 1st run, where NTP system fails on their network.

This has been replaced with systemd-timesyncd, which appears more stable and should resolve the above issues. However, all v6.11 or lower images will need updating to have this system by default.

Fourdee commented 6 years ago

Completed, and, resolved issues with date:

x86_64

~Should not be affected as HW clock~ https://github.com/Fourdee/DietPi/issues/1753#issuecomment-420182483

bios.forceSetupOnce = "TRUE"

Fourdee commented 6 years ago

Rock64:

NanoPi M1/M1+:

MichaIng commented 6 years ago

Actually we switched already to systemd-timesyncd with v6.9.

As time sync should occur on first boot as well before first run update, I am wondering how the error can occur. Maybe something with run_ntpd being called from old dietpi-update but with already new scripts in place. Have to investigate. Hope we can also find a solution for pre v6.9 users, perhaps with a pre-patch that is done before even dietpi-update goes too far, putting new script versions on place.

Rework of emergency patch actually would be it, or updater updating itself in the first place?

Fourdee commented 6 years ago

ASUS TB | Rock64

🈯️ Fixed with commit in this ticket

Driver always reports state unknown:

3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000

ASUS TB 2.0.7

🈴 WiFi can scan, but fails to connect with both DHCP and STATIC. No errors in dmesg

Fourdee commented 6 years ago

@MichaIng

Maybe something with run_ntpd being called from old dietpi-update but with already new scripts in place. Have to investigate. Hope we can also find a solution for pre v6.9 users, perhaps with a pre-patch that is done before even dietpi-update goes too far, putting new script versions on place.

Yep, it needs resolving. If i can replicate it during image updates, i'll start debugging.

6.9

Yeah, but, may as well do the lot while we are here 👍

Fourdee commented 6 years ago

Simple fix for the date issue?

Ok, Issue occurs on 1st run, prior to patching, with:

G_CHECK_URL deb.debian.org

When the date is lower than the cert of the site being checked. This only occured for me, when I manually set the date back a couple of years, even on the v6.0 images. I am unable to replicate a failed timesync as of yet.

Therefore, the EMR patch is pointless.

Regardless of the cause (still unknown, most likely NTP issue with some networks?), the only solution is to redo the images. As the issue occurs pre-patching for some users.

Ok this is strange, v6.0 > v6.14 went fine, after reboot the issue occured:

root@DietPi:~# systemctl status systemd-timesyncd -l
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; disabled; vend
or preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: inactive (dead)
     Docs: man:systemd-timesyncd.service(8)

Nov 03 17:20:15 DietPi systemd[1]: Starting Network Time Synchronization...
Nov 03 17:20:15 DietPi systemd[1]: Started Network Time Synchronization.
Aug 20 23:30:47 DietPi systemd-timesyncd[1413]: Synchronized to time server 193.
150.34.2:123 (0.debian.pool.ntp.org).
Aug 20 23:31:17 DietPi systemd[1]: Stopping Network Time Synchronization...
Aug 20 23:31:17 DietPi systemd[1]: Stopped Network Time Synchronization.
Aug 20 23:31:17 DietPi systemd[1]: Starting Network Time Synchronization...
Aug 20 23:31:17 DietPi systemd[1]: Started Network Time Synchronization.
Aug 20 23:31:17 DietPi systemd-timesyncd[1603]: Synchronized to time server 134.
0.16.1:123 (0.debian.pool.ntp.org).
Aug 20 23:31:19 DietPi systemd[1]: Stopping Network Time Synchronization...
Aug 20 23:31:19 DietPi systemd[1]: Stopped Network Time Synchronization.

Is enabling systemd-timesyncd and removing NTP, rewriting the date to what it was previously with systemd?

And now

[  OK  ] DietPi-Run_ntpd | systemctl restart systemd-timesyncd
Warning: systemd-timesyncd.service changed on disk. Run 'systemctl daemon-reload' to reload units.

v6.0 > v6.13 reboot, date is reset

Croot@DietPi:~# date
Thu Nov  3 17:17:02 GMT 2016
root@DietPi:~# systemctl status systemd-timesyncd -l
Warning: systemd-timesyncd.service changed on disk. Run 'systemctl da
emon-reload' to reload units.
● systemd-timesyncd.service - Network Time Synchronization
   Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; disabled; vend
or preset: enabled)
  Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
           └─disable-with-time-daemon.conf
   Active: inactive (dead)
     Docs: man:systemd-timesyncd.service(8)
Fourdee commented 6 years ago

@MichaIng

🈯️ Time issue resolved with: https://github.com/Fourdee/DietPi/commit/7635d6ccc9383751fe1bbe8d0b87b7a085a85b76

Cause:

Fourdee commented 6 years ago

Rock64, after PREP:

Fine until reboot after resize.

root@DietPi:~# cat /var/tmp/dietpi/logs/fs_partition_resize.log
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help):
Expert command (? for help): Relocating backup data structures to the end of the disk

Expert command (? for help):
Command (? for help): Partition number (1-7):
Command (? for help): Partition number (7-128, default 7): First sector (34-31116254, default = 262144) or {+-}size{KMGTP}: Last sector (262144-31116254, default = 31116254) or {+-}size{KMGTP}: Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): Changed type of partition to 'Linux filesystem'

Command (? for help): Partition number (1-7): Enter name:
Command (? for help):
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): OK; writing new GUID partition table (GPT) to /dev/mmcblk1.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
Filesystem at /dev/mmcblk1p7 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 2
The filesystem on /dev/mmcblk1p7 is now 3856763 (4k) blocks long.
Fourdee commented 6 years ago

Note to self: https://github.com/midwan/amiberry/issues/264#issuecomment-414215685

Fourdee commented 6 years ago

As the original issue which caused this ticket to be created, is now resolved, and, the most popular images have been updated, i'll flag this as low priority.

Millichrome commented 5 years ago

In Issue 2020 you mention that it's possible to manually add the date (I am trying to install the 6.9 image on an ASUS tinker board). What file do I have to edit to manually set the date on the SD card?

Fourdee commented 5 years ago

@Millichrome

What file do I have to edit to manually set the date on the SD card?

You can set the date in file /etc/fake-hwclock.data I believe (untested).

Failing that, if NTP sync fails for you, and you've tried various NTP mirrors in dietpi-config, you'll need to manually apply a date eg:

date +%Y%m%d -s "20180906"

Then re-run the initial setup with:

dietpi-software
Millichrome commented 5 years ago

I log into my Tinker Board DietPi install via SSH. On first connection I get a request to update the default password and run the the update program. The update routine fails with the following error:

https://i.imgur.com/WkcYKSo.jpg

I can't run any commands or do anything once I get this error (at least via SSH, I haven't tried a physical connection).

On Windows 10, I don't see any way to access the "etc" folder (linux partition?). Do I need to install a flavour of desktop linux to edit "/etc/fake-hwclock.data" ?

Thanks for the help.

MichaIng commented 5 years ago

@Millichrome Thanks for the report.

Can't you just select the button (use on keyboard to reach that one) and can run terminal commands then?

Issue is that timesync did not yet update the system clock. You should be able to run systemctl start systemd-timesyncd as another failsafe, as well run: apt install -y fake-hwclock and then run dietpi-software to rerun first run setup (incl. update etc.).

Fourdee commented 5 years ago

I'll update the asus TB image, will resolve this issue. Due to wget without -no-check-certificates prior to ntpd sync check. And fake-hwclock possible missing.

Will revert to 2.0.5 tinker OS due to 2.0.7 WiFi issues.

Fourdee commented 5 years ago

RPI ref: https://github.com/Fourdee/DietPi/issues/2050

Fourdee commented 5 years ago

🈯️ ASUS TB and RPi image redone

Fourdee commented 5 years ago

@Millichrome

ASUS TB image updated, should resolve the issue you are experiencing.

Please download: https://dietpi.com/downloads/images/DietPi_ASUSTB-ARMv7-Stretch.7z

Try this image and let us know if problems are resolved.

Millichrome commented 5 years ago

@MichaIng

This might be an SSH issue or something with my system setup, but I can't run any commands after I get the error. That's why I was trying edit the date on the old 6.9 TB image.

@Fourdee

Awesome! I will test the new image and report back in a few hours.

EDIT: Installation completed fine with the latest TB build. Thanks for the help!

Now to migrating all my applications. Hopefully the network/disk IO is much better than with the B+. :)

Fourdee commented 5 years ago

@Millichrome

Hopefully the network/disk IO is much better than with the B+. :)

Its in a different league :+1:

Fourdee commented 5 years ago

Note to self, following images still remaining:

Fourdee commented 5 years ago

🈯️ RPi needs redoing: https://github.com/Fourdee/DietPi/issues/2087#issuecomment-423743482

Fourdee commented 5 years ago

Only outstanding images to do are Native PC, however, as 99% of those have HW clocks, this issue should not effect those installs.

As such i'll set to low priority for now.

MichaIng commented 5 years ago

@Fourdee How do you create native PC images? I have an old unused notebook and could do that. Anyway had the plan do use it for DietPi dev, also since it has WiFi of course.

But not sure how critical the used base system is and/or if some manual adjustments are needed to ensure/increase compatibility?

Simple practical issue, perhaps dump question:

Okay, whatever I do, however I order drives within BIOS etc., as long as the USB flash drive is attached, at least when detecting disks in installer, it appears as /dev/sda, while the internal SSD appears as /dev/sdb. The installer then configures initramfs to use /dev/sdb1 as root. But when booting into SSD, it is recognized as /dev/sda, while the USB flash is /dev/sdb then, so root dev path is wrong. To fix: I needed to detach the USB flash as fast as the installer is fully loaded. Then SSD is found at /dev/sda. Very strange issue for my impression, a bid unintelligent installer behaviour. It would be better to use UUID here instead of these changing sdX names...

Next unexpected difficulty:

Okay lets go on and see how well DietPi-PREP runs, with WiFi-only, but local console.

@Fourdee How do you create native PC images? Faced the same root device issue when using a USB drive for the installer? Use targeted or generic diver install for the bootloader/initramfs? As far as I read, targeted fits for very most of the systems and generic might be too large for some systems to boot. So not quite clear, but I guess generic should be still best. So the question indeed is if there are any requirements to the creating system besides being x86_64. This also implies if theoretically a VM would also do (but selecting native PC in DietPi-PREP of course) or if Debian installer then would setup something wrong.

Fourdee commented 5 years ago

How do you create native PC images?

For BIOS/CSM:

Then read the USB stick to an image.

Cant remember if we need to change any GRUB settings, been a while.

UEFI:

MichaIng commented 5 years ago

@Fourdee

  • Remove all HDD's from system (ensures GRUB doesnt pick them up)
  • After install, remove Debian USB stick
  • Make sure fstab uses the PARTUUID, eg:

Does initramfs (from final image/system) use the fstab entries from the partition/drive it is loaded from? Then of course editing fstab would resolve it. However since I could not find a way to edit the fstab from within initramfs (could not mount, since there is no "file system" to mount to), this needs to be done from another system. Anyway needed to run zerofree e.g.

If the root partition is not read from fstab, but somehow stored within the initrd/grub, then I indeed needed to detach the Debian installer USB stick once loaded (runs fully in RAM then), which was otherwise recognized as /dev/sda and the destination drive/stick as /dev/sdb. Perhaps this also depends on the IDE/SATA/USB controllers, in which order they recognize the drives.

Fourdee commented 5 years ago

@MichaIng

Does initramfs (from final image/system) use the fstab entries from the partition/drive it is loaded from?

Actually, ignore everything I just said in regards to PARTUUID, UUID will work fine, that is what update-grub will use.

Can check what gets applied with: /boot/grub/grub.cfg

menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-aa010bdf-3471-4ca4-ab$
        load_video
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root  aa010bdf-3471-4ca4-ab17-d1488d382949
        else
          search --no-floppy --fs-uuid --set=root aa010bdf-3471-4ca4-ab17-d1488d382949
        fi
        echo    'Loading Linux 4.15.0-0.bpo.2-amd64 ...'
        linux   /boot/vmlinuz-4.15.0-0.bpo.2-amd64 root=UUID=aa010bdf-3471-4ca4-ab17-d1488d382949 ro net.ifnames=0 consoleblank=0 quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-4.15.0-0.bpo.2-amd64
}
Fourdee commented 5 years ago

REF:

MichaIng commented 5 years ago

@Fourdee I think the RPi image needs to be redone. It's on DietPi v6.17 where I added a bug to exit, if hardware identifier file could not be found, which is meant to not exist on RPi: https://github.com/Fourdee/DietPi/commit/eae7af9d50775d833c2c672a344f364df1b02ae5#diff-c7bcfe6e7ece17b798b4d08a37df308cR37

You fixed that with v6.18: https://github.com/Fourdee/DietPi/commit/0c0995312500865d049c13276032fb94bfb43f89#diff-c7bcfe6e7ece17b798b4d08a37df308c

Most likely related: https://dietpi.com/phpbb/viewtopic.php?f=11&t=5463&p=16292#p16292 I am wondering that we didn't get more reports 😅.

If you are busy, I can do that on my RPi2. Would be mind to link your image finalize steps? I always forget where you posted that already. However zerofree and dd should be it. Can the img file be created from SDcard on Windows machine as well or do you always create it on a Linux system?

Fourdee commented 5 years ago

@MichaIng

If I remember and can, i'll redo the image after v6.20 release.

Heres the finalisation script:

#!/bin/bash
{

    #check packages are installed
    if (( ! $(dpkg -l | grep -ci -m1 'gparted') )); then

        apt-get install gparted gdisk gpart zerofree -y

    fi

    #-------------------------------------------------------------------------------
    #MUST LEAVE 50MB+ space or automation autologin may fail due to 0 free space!!!
    #-------------------------------------------------------------------------------

    RK_SINGLE_IMG=0

    CP_PARTITIONS=0

    #fdisk -l -u "$IMAGE_FP/$IMAGE_NAME"

    IMAGE_FP='/mnt/dietpi_userdata/#Backups/_Daniel/Projects/DietPi'

    IMAGE_NAME='DietPi_v6.19_RockPro64-ARMv8-Stretch'
    IMAGE_NAME+='.img'

    INDEX_BOOT_PART=1
    INDEX_ROOTFS_PART=2
    if (( $RK_SINGLE_IMG )); then

        INDEX_ROOTFS_PART=1

    fi

#GPT images (ROCKPro 64) after reading the image AND again after shrinking
#to fix:
#   GPT PMBR size mismatch (4458495 != 15523839) will be corrected by w(rite).
#   15523806
#RUN
# gdisk "$IMAGE_FP/$IMAGE_NAME"
#   w | y
#

    if [[ ! -f $IMAGE_FP'/'$IMAGE_NAME ]]; then

        read -p "No file found..."
        exit

    fi

    #resize2fs /dev/loop2p7 500M
    #dumpe2fs -h /dev/loop2p7 | grep 'Block count'
    #Block count * Block size / 1024 / 1024 =MB?

    modprobe loop
    losetup -f
    losetup /dev/loop2 "$IMAGE_FP/$IMAGE_NAME"
    partprobe /dev/loop2

    if (( $RK_SINGLE_IMG )); then

        e2fsck -f /dev/loop2

    else

        if [[ $INDEX_BOOT_PART ]]; then

            e2fsck -f /dev/loop2p$INDEX_BOOT_PART

        fi
        e2fsck -f /dev/loop2p$INDEX_ROOTFS_PART

    fi

    #Optional, copy
    if (( $CP_PARTITIONS )); then

        mkdir -p /mnt/loop2p1
        mkdir -p /mnt/loop2p2
        mount /dev/loop2p$INDEX_BOOT_PART /mnt/loop2p1
        mount /dev/loop2p$INDEX_ROOTFS_PART /mnt/loop2p2

        echo -e "\ncopying data to $HOME\n"
        rm -R $HOME/loop2p1
        rm -R $HOME/loop2p2

        cp -R /mnt/loop2p1 $HOME/
        cp -R /mnt/loop2p2 $HOME/
        rm -R $HOME/loop2p1/lost+found
        rm -R $HOME/loop2p2/lost+found

    fi

    sync

    umount /mnt/loop2p1
    umount /mnt/loop2p2
    umount /mnt/loop2

    #Resize partitions
    echo -e "\Resize partitions\n"
    gparted /dev/loop2

    sync

    #Mount for file changes, if required
    mkdir -p /mnt/loop2p1
    mkdir -p /mnt/loop2p2
    if (( $RK_SINGLE_IMG )); then

        mount /dev/loop2 /mnt/loop2p1

    else

        if [[ $INDEX_BOOT_PART ]]; then

            mount /dev/loop2p$INDEX_BOOT_PART /mnt/loop2p1

        fi
        mount /dev/loop2p$INDEX_ROOTFS_PART /mnt/loop2p2

    fi

    if (( $CP_PARTITIONS )); then

        echo -e "\nCopying data back\n"
        cp -R $HOME/loop2p1/* /mnt/loop2p1/
        cp -R $HOME/loop2p2/* /mnt/loop2p2/

    fi

    read -p "Partitions mounted to /mnt/loop2px, make changes to files if required..."

    sync

    umount /mnt/loop2p1
    umount /mnt/loop2p2

    #Resize partitions
    # echo -e "Resize partitions"
    # gparted /dev/loop2

    # sync

    read -p "If failed press CTRL+C to exit and prevent further action, else, press any key to continue..."

    if (( $RK_SINGLE_IMG )); then

        zerofree -v /dev/loop2

    else

        for (( i=0; i<10; i++ ))
        do

            if [[ -b /dev/loop2p$i ]]; then

                zerofree -v /dev/loop2p$i

            fi

        done

    fi

    losetup -d /dev/loop2

    if (( $RK_SINGLE_IMG )); then

        END_PARTITION='850M'

    else

        # - debug
        # END_PARTITION="$(echo p | parted "$IMAGE_FP/$IMAGE_NAME" | grep ' 1 ' | awk '{print $3}')"

        if (( $INDEX_ROOTFS_PART == 1 )); then

            END_PARTITION=$(( $(fdisk -l "$IMAGE_FP/$IMAGE_NAME" | grep ".img$INDEX_ROOTFS_PART" | awk '{print $4}') + 1 ))

        else

            END_PARTITION=$(( $(fdisk -l "$IMAGE_FP/$IMAGE_NAME" | grep ".img$INDEX_ROOTFS_PART" | awk '{print $3}') + 1 ))

        fi

        END_PARTITION=$(( 512 * ( $END_PARTITION + 256 ) )) #64 bytes for secondary GPT + safety net

    fi

    truncate --size=$END_PARTITION "$IMAGE_FP/$IMAGE_NAME"

    echo -e "\nCleaning up $HOME/loop2px\n"
    rm -R $HOME/loop2p1
    rm -R $HOME/loop2p2

    read -p "Done, press any key to continue..."

    exit

}
MichaIng commented 5 years ago

@Fourdee Nice, maybe we can add this to .meta, so we can transparently work on/update it as well and available for end users for own image creations.

Fourdee commented 5 years ago

ToDo, update:

MichaIng commented 5 years ago

@Fourdee May I add a Odroid C1 Stretch image?

More and more issues appear with Jessie not offering required library/package versions, e.g. now Emby: https://github.com/Fourdee/DietPi/issues/2521

Btw great work to update so much images to v6.20, I recognized meanwhile. That as well prevented many users from running into first run update loop due to the updater issue 👍!!

Fourdee commented 5 years ago

@MichaIng

May I add a Odroid C1 Stretch image?

Please, I still unable to find mine, most likely in a toy box somewhere (kids lol).

  • I know this means a distro upgrade since Meveric does not offer a Stretch image. However if this can succeed, then via DietPi-PREP

Yep, agree Stretch is preferred. However, software from Meverics APT repo (kernel/kodi etc) may not work.

Although, if above is an issue, update to latest kernel, then remove his APT repo from /etc/apt/sources.list.d

MichaIng commented 5 years ago

@Fourdee Ah sorry, I have not Odroid C1 here either 😢. I meant adding it to the ToDo list 😉.

Hmm, kernel packages should work independently from Debian version, at least on RPi this was never an issue (Stretch kernel on Buster, and that time Jessie kernel on Stretch). But of course needs testing. Are Meverics other repos device specific and for C1 only Jessie available? 🤔 Okay that would be indeed an issue. I mean we can anyway not support it forever, but loosing Kodi and certain media software support is quite a nasty bumper...

From what I can see here it looks fine. C1 specific packages (and kernel) do not require Jessie. All dependencies are >=, non <= where a Stretch package could be too new: http://fuzon.co.uk/meveric/dists/all/c1/binary-armhf/Packages The software repo source can be switch from jessie to stretch. This is actually something we could/should do automatically within DietPi-PREP based on chosen distro version: https://github.com/Fourdee/DietPi/blob/master/PREP_SYSTEM_FOR_DIETPI.sh#L578

Related new issue: https://github.com/Fourdee/DietPi/issues/2561

MichaIng commented 5 years ago

Other outdated images

I will do the x86_64 BIOS image the next days, UEFI is currently outside of my possibilities.

MichaIng commented 5 years ago

Updated VirtualBox Stretch image: https://dietpi.com/downloads/images/DietPi_VirtualBox-x86_64-Stretch.7z

Updated VirtualBox Buster image: https://dietpi.com/downloads/images/DietPi_VirtualBox-x86_64-Buster.7z

MichaIng commented 5 years ago

Updated VirtualBox Stretch image: https://dietpi.com/downloads/images/DietPi_VMware-x86_64-Stretch.7z

Updated VirtualBox Buster image: https://dietpi.com/downloads/images/DietPi_VMware-x86_64-Buster.7z

Links on homepage have been updated as well (manually), since old images are zip instead of 7z.

MichaIng commented 5 years ago

Updated NativePC BIOS image ready for testing: ~https://dietpi.com/downloads/testing/DietPi_NativePC-BIOS-x86_64-Stretch.7z~

MichaIng commented 5 years ago

Updated NativePC BIOS image released: https://dietpi.com/downloads/images/DietPi_NativePC-BIOS-x86_64-Stretch.7z

MichaIng commented 5 years ago

Updated RPi image ready for testing: https://dietpi.com/downloads/testing/DietPi_RPi-ARMv6-Stretch.7z

MichaIng commented 5 years ago

Updated RPi image with yesterday released kernel v4.19 + FIRST Raspbian Buster image available for testing: https://dietpi.com/downloads/testing/

MichaIng commented 5 years ago

I close this in favour of: https://github.com/MichaIng/DietPi/issues/2979