search

1000001101000 / Debian_on_Buffalo

Tools for Installing/Running Debian on Buffalo ARM based Linkstation/Terastation/Kurobox/Cloudstor devices.
333 stars 41 forks source link

Crash on Terastation Pro II #64

Open DavidDah opened 4 years ago

DavidDah commented 4 years ago

I'm not sure is this installer problem or partition manager issue, however after reformating partitions and creating software raid 10, md1 device started resyncing and then this problem occurred and terastation rebooted. I have kept existing partitions from stock firmware for /boot, just deleted everything else and created swap and another partition for / filesystem and tried to create RAID 10. I have used latest initrd.buffalo and uImage.buffalo from [(https://github.com/1000001101000/Debian_on_Buffalo/tree/master/Buster/installer_images/armel_devices)]

`[ 1 installer 2 shell 3- shell (4*log) ][ Oct 30 23:31 ] Oct 30 23:30:15 net/hw-detect.hotplug: Detected hotpluggable network interface eth0 Oct 30 23:30:15 net/hw-detect.hotplug: Detected hotpluggable network interface lo Oct 30 23:30:18 kernel: [ 720.909371] Adding 155644k swap on /dev/sda2. Priority:-1 extents:1 across:155644k FS Oct 30 23:30:18 kernel: [ 720.950322] Adding 155644k swap on /dev/sdb2. Priority:-2 extents:1 across:155644k FS Oct 30 23:30:18 kernel: [ 720.990655] Adding 155644k swap on /dev/sdc2. Priority:-3 extents:1 across:155644k FS Oct 30 23:30:18 kernel: [ 721.031660] Adding 155644k swap on /dev/sdd2. Priority:-4 extents:1 across:155644k FS Oct 30 23:30:52 partman-md: Set the sync speed for RAID devices to 1000 Oct 30 23:30:52 partman-md: Selected spare count: 0 Oct 30 23:30:52 partman-md: RAID devices count: 4 Oct 30 23:30:52 partman-md: Spare devices count: 0 Oct 30 23:30:52 partman-md: mdadm: /dev/sda3 appears to contain an ext2fs file system Oct 30 23:30:52 partman-md: size=487931904K mtime=Thu Jan 1 00:00:00 1970 Oct 30 23:30:52 partman-md: mdadm: /dev/sdb3 appears to contain an ext2fs file system Oct 30 23:30:52 partman-md: size=487931904K mtime=Thu Jan 1 00:00:00 1970 Oct 30 23:30:52 partman-md: mdadm: /dev/sdc3 appears to contain an ext2fs file system Oct 30 23:30:52 partman-md: size=487931904K mtime=Thu Jan 1 00:00:00 1970 Oct 30 23:30:52 partman-md: mdadm: /dev/sdd3 appears to contain an ext2fs file system Oct 30 23:30:52 partman-md: size=487931904K mtime=Thu Jan 1 00:00:00 1970 Oct 30 23:30:53 kernel: [ 755.716789] md/raid10:md1: not clean -- starting background reconstruction Oct 30 23:30:53 kernel: [ 755.716805] md/raid10:md1: active with 4 out of 4 devices Oct 30 23:30:53 partman-md: mdadm: array /dev/md/1 started. Oct 30 23:30:53 kernel: [ 755.787257] md1: detected capacity change from 0 to 999283490816 Oct 30 23:30:53 kernel: [ 755.791310] md: resync of RAID array md1 Oct 30 23:31:25 kernel: [ 788.212918] ------------[ cut here ]------------ Oct 30 23:31:25 kernel: [ 788.213098] WARNING: CPU: 0 PID: 7 at drivers/scsi/scsi_lib.c:1067 scsi_init_io+0x58/0x1c4 [scsi_mod] Oct 30 23:31:25 kernel: [ 788.213105] Modules linked in: raid10 dm_mod dax raid1 md_mod xfs libcrc32c vfat fat nls_base ext4 crc16 jbd2 crc32c_generic fscrypto ecb mbcache ipv6 af_packet marvell mvmdio mv643xx_eth sata_mv libata sd_mod scsi_mod leds_gpio Oct 30 23:31:25 kernel: [ 788.213239] CPU: 0 PID: 7 Comm: ksoftirqd/0 Not tainted 4.12.13 #3 Oct 30 23:31:25 kernel: [ 788.213247] Hardware name: Buffalo Terastation Pro II/Live Oct 30 23:31:25 kernel: [ 788.213291] [] (unwind_backtrace) from [] (show_stack+0x18/0x1c) Oct 30 23:31:25 kernel: [ 788.213323] [] (show_stack) from [] (warn+0xe8/0x104) Oct 30 23:31:25 kernel: [ 788.213352] [] (warn) from [] (warn_slowpath_null+0x24/0x2c) Oct 30 23:31:25 kernel: [ 788.213499] [] (warn_slowpath_null) from [] (scsi_init_io+0x58/0x1c4 [scsi_mod]) Oct 30 23:31:25 kernel: [ 788.213664] [] (scsi_init_io [scsi_mod]) from [] (sd_init_command+0x4f4/0xd90 [sd_mod]) Oct 30 23:31:25 kernel: [ 788.213821] [] (sd_init_command [sd_mod]) from [] (scsi_prep_fn+0x7c/0x104 [scsi_mod]) Oct 30 23:31:25 kernel: [ 788.213951] [] (scsi_prep_fn [scsi_mod]) from [] (blk_peek_request+0x11c/0x23c) Oct 30 23:31:25 kernel: [ 788.214081] [] (blk_peek_request) from [] (scsi_request_fn+0x28/0x6cc [scsi_mod]) Oct 30 23:31:25 kernel: [ 788.214207] [] (scsi_request_fn [scsi_mod]) from [] (blk_run_queue+0x3c/0x4c) Oct 30 23:31:25 kernel: [ 788.214235] [] (blk_run_queue) from [] (blk_run_queue+0x1c/0x24) Oct 30 23:31:25 kernel: [ 788.214356] [] (blk_run_queue) from [] (scsi_run_queue+0x274/0x278 [scsi_mod]) Oct 30 23:31:25 kernel: [ 788.214567] [] (scsi_run_queue [scsi_mod]) from [] (scsi_end_request+0x148/0x150 [scsi_mod]) Oct 30 23:31:25 kernel: [ 788.214775] [] (scsi_end_request [scsi_mod]) from [] (scsi_io_completion+0x244/0x524 [scsi_mod]) Oct 30 23:31:25 kernel: [ 788.214897] [] (scsi_io_completion [scsi_mod]) from [] (blk_done_softirq+0x98/0xa0) Oct 30 23:31:25 kernel: [ 788.214929] [] (blk_done_softirq) from [] (__do_softirq+0x1d4/0x2bc) Oct 30 23:31:25 kernel: [ 788.214961] [] (__do_softirq) from [] (run_ksoftirqd+0x2c/0x5c) Oct 30 23:31:25 kernel: [ 788.214995] [] (run_ksoftirqd) from [] (smpboot_thread_fn+0x178/0x180) Oct 30 23:31:25 kernel: [ 788.215030] [] (smpboot_thread_fn) from [] (kthread+0x124/0x13c) Oct 30 23:31:25 kernel: [ 788.215061] [] (kthread) from [] (ret_from_fork+0x14/0x24) Oct 30 23:31:25 kernel: [ 788.215073] ---[ end trace a43daefba01e5af5 ]---

`

DavidDah commented 4 years ago

It's Terastation PRO II device with 4 x 500GB drives.

1000001101000 commented 4 years ago

Good to hear form you!

I don't believe I've seen that before. Could you try again and see if it happens the same way? When you do, could you take a screen shot of your partitioning screen? I can try the same settings on mine and see if I can re-create it.

another option is to try the new process I put together (and need to finish documenting and start pushing more) which generates a disk image rather than running the installer on the device. There are pro's and con's to each method but sidestepping weird things happening during the install on older devices is the main advantage. That process can be found here: https://github.com/1000001101000/Debian_on_Buffalo/wiki/Alternate-install-method-via-debootstrap-script

DavidDah commented 4 years ago

Thank You for fast replay! Unfortunately I was unable to make screenshots since this time terastation rebooted before it showed partition screen with the same message in the log. However at the moment I'm syncing raid on my PC and when it's finished I will put drives back into TS and try to continue with installation process. If it fails for any reason I will try with bootstrap script and I will update you here about the progress of installation, I do have another TS Live and if TS2 pro install normally, I will test bootstrap on TS Live and see how it works.

DavidDah commented 4 years ago

Just side note here stock firmware partition layout is not compatible with installer, stock firmware partition layout is /boot 290MB / 490MB swap 128 MB and rest for XFS partition, which is not enough to install debian since / partition requires more than 1 GB. So it's necessary to repartition drives in order to make enough free space on / partition for debian installation.

1000001101000 commented 4 years ago

I meant to add a note about that on the wiki, I'm not sure which models need to be repartitioned but imagine it's all of them from that generation.

I could imagine attempting to resync an array could cause problems during an install. There are some commands you could run that slow down or disable the resync process which might be a good idea during the install. I might even be able to make the installer do that automatically in the future.

I haven't heard of this happening before but I also have no idea how many folks have actually created raid arrays using the installer.

DavidDah commented 4 years ago

Maybe i could make documentation for installing on older TS devices since i still have few of them laying around and You can review it and add it to the project. Regarding partitioning and setting up RAID on older devices it's really painful experience. It takes 5-10 seconds each time when you press button to do any action in the partitioner and obviously syncing RAID on this old devices cause device crash. I was thinking about slowing down sync trough procfs but somehow I thought it will be faster to sync RAID on PC and put back disks to TS. Even on PC it takes about 5-6h to sync RAID 10 on 4 drives since this are slow HDDs by today's standards.

Question is how can we speedup this whole partitioning and RAID creation stuff and let the installation proceed normally, maybe froze sync and let it run after Debian is booted?

1000001101000 commented 4 years ago

That would be great!

What I would recommend is to set up /boot, swap and / and leave creating the data array for after the install is complete. That should avoid any issues caused by the sync running alongside the installer.

The debootstrap/disk image method follows that same strategy by setting up the system arrays with just the one partition each so that you can extend it as desired once you've got Debian up and running.

DavidDah commented 4 years ago

This terastation is proving to be hard nut to crack, I have created new partitions on PC /boot 1 GB, swap 2GB, / 10GB and put / partition into raid 10 and synced raid on PC, then i formatted it to ext4. /boot partition was left as is from stock rom with ext3 on it.

I verified that /boot is empty since I wanted to push initrd and uImage from TFTP after reboot. As far as I understand process TS Pro v2 will start in EM mode with TFTP pulling uImage.buffalo and initrd.buffalo from TFTP server on 192.168.11.1 and this is exactly what happened.

Terastation booted from provided images, installer proceeded normally, i have mounted /boot, swap and / filesystem and left rest of free space for latter. Everything else worked and about 40 minutes latter i was running debian. I have created /dev/md2 from rest of free space and mounted it and it's now doig raid sync which should last for another 5-6h.

Once it's done I will revert back TS to stock FS and make guide step by step how to install debian on TS 2 pro.

1000001101000 commented 4 years ago

Nice!

DavidDah commented 4 years ago

Just side note here, after install NTFS driver for linux fuse-3g terastation crashed and went into bootloop.

1000001101000 commented 4 years ago

Take a look at the initrd.buffalo generated after you installed that. It might have pushed the size past the ~7mb limit.

If so there are things we can do to work around it

DavidDah commented 4 years ago

I have wrote guide about installing Debian on TS PRO 2 however after installer is finished TS reboots normally and then starts in EM mode again. I suspect that I maybe made some mistake with creating partitions but I'm not sure.

Could you please take look at it https://github.com/DavidDah/Debian_on_Buffalo/wiki/How-to-install-Debian-on-TeraStation-PRO-II-TS-HTGL-R5 and try to figure out what i did wrong.

Once we figure it out You can merge that guide into project wiki. Thank You very much.

P.S. This is last lines from installer before the end of the installation.

Nov 1 04:45:47 in-target: ^M Nov 1 04:45:47 in-target: Unpacking flash-kernel (3.99) ... Nov 1 04:45:47 in-target: ^M Nov 1 04:45:48 in-target: Setting up liblzo2-2:armel (2.10-0.1) ... Nov 1 04:45:48 in-target: ^M Nov 1 04:45:49 in-target: Setting up mtd-utils (1:2.0.1-1) ... Nov 1 04:45:49 in-target: ^M Nov 1 04:45:49 in-target: Setting up devio (1.2-1.2+b1) ... Nov 1 04:45:49 in-target: ^M Nov 1 04:45:49 in-target: Setting up flash-kernel (3.99) ... Nov 1 04:45:49 in-target: ^M Nov 1 04:45:53 in-target: debconf: unable to initialize frontend: Passthrough Nov 1 04:45:53 in-target: ^M Nov 1 04:45:53 in-target: debconf: (Failed to open fd 3: Bad file descriptor at (eval 18) line 3.) Nov 1 04:45:53 in-target: ^M Nov 1 04:45:53 in-target: debconf: falling back to frontend: Noninteractive Nov 1 04:45:53 in-target: ^M Nov 1 04:45:55 in-target: ^M Nov 1 04:45:55 in-target: Creating config file /etc/default/flash-kernel with new version Nov 1 04:45:55 in-target: ^M Nov 1 04:45:56 in-target: Processing triggers for man-db (2.8.5-2) ... Nov 1 04:45:56 in-target: ^M Nov 1 04:45:59 in-target: Processing triggers for libc-bin (2.28-10) ... Nov 1 04:45:59 in-target: ^M Nov 1 04:46:06 in-target: update-initramfs: Generating /boot/initrd.img-4.12.13 Nov 1 04:46:59 in-target: flash-kernel: installing version 4.12.13 Nov 1 04:47:23 in-target: Generating kernel u-boot image... Nov 1 04:47:23 in-target: done. Nov 1 04:47:23 in-target: Taking backup of uImage.buffalo. Nov 1 04:47:23 in-target: Installing new uImage.buffalo. Nov 1 04:47:23 in-target: Generating initramfs u-boot image... Nov 1 04:47:24 in-target: done. Nov 1 04:47:24 in-target: Taking backup of initrd.buffalo. Nov 1 04:47:24 in-target: Installing new initrd.buffalo. Nov 1 04:47:25 finish-install: info: Running /usr/lib/finish-install.d/07speakup Nov 1 04:47:25 finish-install: info: Running /usr/lib/finish-install.d/10clock-setup Nov 1 04:47:25 anna-install: Installing rtc-modules Nov 1 04:47:25 hw-detect: modprobe: ERROR: could not open builtin file '/lib/modules/4.12.13/modules.builtin.bin' Nov 1 04:47:25 hw-detect: modprobe: FATAL: Module rtc not found in directory /lib/modules/4.12.13 Nov 1 04:47:25 clock-setup: rtc module not loaded Nov 1 04:47:25 hw-detect: modprobe: ERROR: could not open builtin file '/lib/modules/4.12.13/modules.builtin.bin' Nov 1 04:47:25 hw-detect: modprobe: FATAL: Module rtc-dev not found in directory /lib/modules/4.12.13 Nov 1 04:47:25 clock-setup: rtc-dev module not loaded Nov 1 04:47:25 clock-setup: hwclock: Nov 1 04:47:25 clock-setup: use --verbose, --debug has been deprecated. Nov 1 04:47:25 clock-setup: Nov 1 04:47:26 finish-install: info: Running /usr/lib/finish-install.d/10update-initramfs Nov 1 04:47:27 finish-install: dpkg-divert: warning: diverting file '/sbin/start-stop-daemon' from an Essential package with rename is dangerous, use --no-rename Nov 1 04:47:28 finish-install: info: Running /usr/lib/finish-install.d/20final-message Nov 1 04:49:01 finish-install: info: Running /usr/lib/finish-install.d/30hw-detect Nov 1 04:49:01 finish-install: info: Running /usr/lib/finish-install.d/50config-target-network Nov 1 04:49:01 finish-install: info: Running /usr/lib/finish-install.d/55netcfg-copy-config Nov 1 04:49:02 finish-install: dpkg-divert: warning: diverting file '/sbin/start-stop-daemon' from an Essential package with rename is dangerous, use --no-rename Nov 1 04:49:03 finish-install: info: Running /usr/lib/finish-install.d/60cleanup Nov 1 04:49:04 finish-install: info: Running /usr/lib/finish-install.d/65partman-md Nov 1 04:49:04 finish-install: info: Running /usr/lib/finish-install.d/70mtab Nov 1 04:49:04 finish-install: info: Running /usr/lib/finish-install.d/90base-installer Nov 1 04:49:04 finish-install: info: Running /usr/lib/finish-install.d/90console Nov 1 04:49:04 finish-install: Configuring init for serial console Nov 1 04:49:04 finish-install: info: Running /usr/lib/finish-install.d/94random-seed Nov 1 04:49:04 finish-install: info: Running /usr/lib/finish-install.d/94save-logs Nov 1 04:49:05 finish-install: info: Running /usr/lib/finish-install.d/95umount

1000001101000 commented 4 years ago

Did you boot over tftp to get in the installer?

My guess is using raid10 /boot wont work. I’d strongly recommend raid1 for /boot at least.

DavidDah commented 4 years ago

No I didn't use TFTP, I created raid 10, formated it to ext3 and downloaded installer to /boot while drives were still connected to PC, after that I put back drives to TS and installer started automatically from /boot.

DavidDah commented 4 years ago

No luck with raid 1 on /boot, i have added sda1 and sdb1 to raid1, downloaded uimage and initrd directly on boot, installer started but TS rebooted while initializing partmanager. Im gonna restore stock firmware and check exactly how partitioning is done.

1000001101000 commented 4 years ago

rebooting early in installer might indicate the code that disables the startup watchdog timer. a simple reboot might resolve that.

a few things to check: is the initrd > 7mb? shouldn't happen from the installer but adding some app automatically add initrd hooks that make it grow. make sure the partition table isn't gpt. installer shouldn't do that unless you use disks > 2tb make sure /boot isn't too big (<=768mb). it seemed fine before. make sure /boot is ext3

DavidDah commented 4 years ago

You were right, /boot and "/" are RAID 1 in stock firmware

root@TS-HTGLA42:/mnt/array1# mdadm -Q --detail /dev/md0 /dev/md0: Version : 00.90.03 Creation Time : Mon Nov 2 08:02:55 2020 Raid Level : raid1 Array Size : 297088 (290.17 MiB 304.22 MB) Device Size : 297088 (290.17 MiB 304.22 MB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent

Update Time : Sun Nov  1 23:28:39 2020
      State : clean

Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0

       UUID : 2f03348e:56452ff2:9e0daaa8:909f6fdb
     Events : 0.174

Number   Major   Minor   RaidDevice State
   0       8        1        0      active sync   /dev/sda1
   1       8       17        1      active sync   /dev/sdb1
   2       8       33        2      active sync   /dev/sdc1
   3       8       49        3      active sync   /dev/sdd1

root@TS-HTGLA42:/mnt/array1# mdadm -Q --detail /dev/md1 /dev/md1: Version : 00.90.03 Creation Time : Mon Nov 2 08:02:56 2020 Raid Level : raid1 Array Size : 497920 (486.33 MiB 509.87 MB) Device Size : 497920 (486.33 MiB 509.87 MB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 1 Persistence : Superblock is persistent

Update Time : Sun Nov  1 23:36:40 2020
      State : clean

Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0

       UUID : ee090f83:3d580ca9:a818aa3d:42dd70da
     Events : 0.966

Number   Major   Minor   RaidDevice State
   0       8        2        0      active sync   /dev/sda2
   1       8       18        1      active sync   /dev/sdb2
   2       8       34        2      active sync   /dev/sdc2
   3       8       50        3      active sync   /dev/sdd2

root@TS-HTGLA42:/mnt/array1#

root@TS-HTGLA42:/mnt/array1# cat /etc/fstab /dev/root / ext3 defaults 1 1 proc /proc proc defaults 0 0 devpts /dev/pts devpts gid=4,mode=620 0 0

root@TS-HTGLA42:/mnt/array1# mount rootfs on / type rootfs (rw) /dev/root on / type xfs (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw) /proc/bus/usb/ on /proc/bus/usb type usbfs (rw) /dev/loop1 on /mnt type ext2 (ro,nogrpid) /dev/ram1 on /mnt/ram type tmpfs (rw) /dev/md0 on /boot type ext3 (rw,data=ordered) /dev/md2 on /mnt/array1 type xfs (rw,noatime,sunit=128,swidth=512)

root@TS-HTGLA42:/mnt/array1# fdisk -l

Disk /dev/sda: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System /dev/sda1 1 37 297171 83 Linux /dev/sda2 38 99 498015 83 Linux /dev/sda4 100 60801 487588815 5 Extended /dev/sda5 100 116 136521 82 Linux swap /dev/sda6 117 60784 487315678+ 83 Linux

Disk /dev/sdb: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System /dev/sdb1 1 37 297171 83 Linux /dev/sdb2 38 99 498015 83 Linux /dev/sdb4 100 60801 487588815 5 Extended /dev/sdb5 100 116 136521 82 Linux swap /dev/sdb6 117 60784 487315678+ 83 Linux

Disk /dev/sdc: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System /dev/sdc1 1 37 297171 83 Linux /dev/sdc2 38 99 498015 83 Linux /dev/sdc4 100 60801 487588815 5 Extended /dev/sdc5 100 116 136521 82 Linux swap /dev/sdc6 117 60784 487315678+ 83 Linux

Disk /dev/sdd: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System /dev/sdd1 1 37 297171 83 Linux /dev/sdd2 38 99 498015 83 Linux /dev/sdd4 100 60801 487588815 5 Extended /dev/sdd5 100 116 136521 82 Linux swap /dev/sdd6 117 60784 487315678+ 83 Linux

root@TS-HTGLA42:/mnt/array1# mdadm --detail -scan --verbose ARRAY /dev/md0 level=raid1 num-devices=4 UUID=2f03348e:56452ff2:9e0daaa8:909f6fdb devices=/dev/sda1,/dev/sdb1,/dev/sdc1,/dev/sdd1 ARRAY /dev/md1 level=raid1 num-devices=4 UUID=ee090f83:3d580ca9:a818aa3d:42dd70da devices=/dev/sda2,/dev/sdb2,/dev/sdc2,/dev/sdd2 ARRAY /dev/md2 level=linear num-devices=4 UUID=fbc20df7:a5ba67fc:bfee4477:89e26673 devices=/dev/sda6,/dev/sdb6,/dev/sdc6,/dev/sdd6

DavidDah commented 4 years ago

I managed to install Debian this time by using RAID 1, however there are 2 issues.

  1. When booting installer from /boot partition after being copied on PC (No TFTP) it seems that micro-evtd scripts are not initiated, boot done signal is not sent and TS reboots after 10 minutes, also LCD doesn't display Debian ARM installer message. I'v managed to overcome this by starting shell from install before it rebooted TS and executing manually

/source/micro-evtd -s 0003

  1. After installing Debian, running update-initramfs will break the system and send TS into bootloop, some debian packages like fuse-3g will run update-initramfs and this will leave system broken. Is there any solution to update-initramfs problem?
1000001101000 commented 4 years ago

1 might be my fault, I've done some work on micro-evtd lately. I've tested those changes on a ts2pro before committing them but haven't actually tested them inside the installer. The automation I've set up automatically generates new installer images when a new micro-evtd version gets commited. I'll fix that as soon as I get a chance.

for #2 there are a few things you can do to minimize initrd size, specifically using xz compression. I've got details in this post: https://github.com/1000001101000/Debian_on_Buffalo/issues/62#issuecomment-713225833

At one point I intended to make those settings the default, I don't recall why I haven't. maybe I'll throw that in along with any micro-evtd fix.

DavidDah commented 4 years ago

TS PRO II has 128MB ram, could I try increasing run size or u-boot limit will kick in ? root@tspro2:/etc/initramfs-tools/conf.d# free -h total used free shared buff/cache available Mem: 121Mi 86Mi 3.0Mi 1.0Mi 31Mi 28Mi

`cat /etc/initramfs-tools/scripts/init-bottom/runsize.sh

run_size="$(busybox df -m /run | busybox tail -n 1 | busybox awk '{print $2}')"

mount -o remount,nosuid,noexec,size=26M,nr_inodes=4096 /run`

Also I have tried adding options to /etc/initramfs-tools/conf.d/compress and managed to significantly reduce kernel size about 30% root@tspro2:/var/log# ls -al /boot/ total 23924 drwxr-xr-x 4 root root 4096 Nov 2 23:29 . drwxr-xr-x 18 root root 4096 Nov 2 04:06 .. drwxr-xr-x 2 root root 4096 Nov 2 05:56 backup -rw-r--r-- 1 root root 164677 Oct 13 2019 config-4.12.13 -rw-r--r-- 1 root root 4871276 Nov 2 23:29 initrd.buffalo -rw-r--r-- 1 root root 6746933 Nov 2 21:19 initrd.buffalo.bak -rw-r--r-- 1 root root 4871212 Nov 2 23:28 initrd.img-4.12.13 drwx------ 2 root root 16384 Nov 2 02:41 lost+found -rw-r--r-- 1 root root 1549363 Oct 13 2019 System.map-4.12.13 -rw-r--r-- 1 root root 2063608 Nov 2 23:29 uImage.buffalo -rw-r--r-- 1 root root 2063608 Nov 2 21:19 uImage.buffalo.bak -rwxr-xr-x 1 root root 2063536 Oct 13 2019 vmlinuz-4.12.13

1000001101000 commented 4 years ago

runsize won't cause any problems for uboot.

those errors seem odd to me, did you put the lines in quotes or something like that?

DavidDah commented 4 years ago

Yes i have put quotes that was the issue, however after reducing size of kernel for about 30%, TS went into bootloop again.. Do You have any advice how to debug this issue?

-rw-r--r-- 1 root root 4871276 Nov 2 23:29 initrd.buffalo -rw-r--r-- 1 root root 6746933 Nov 2 21:19 initrd.buffalo.bak

1000001101000 commented 4 years ago

if you send this command the serial port on the back becomes a serial console:

micro-evtd -s 000f

it stays that way through a reboot but not a poweroff. oddly enough i have this built into the installer though it probably isn't working if micro-evtd isn't working.

You can debug just about anything if you get a usb->serial adapter and connect it. It's too bad none of the newer models included that feature.

DavidDah commented 4 years ago

Excellent!!! Does it require straight serial cable ?

1000001101000 commented 4 years ago

I use something similar to this:

https://www.amazon.com/Serial-Adapter-Prolific-Chipset-Windows/dp/B0753HBT12/

DavidDah commented 4 years ago

Iv managed to connect with serial cable to terastation (I have large collection of serial cables, maybe i could open museum of serial cables soon). Here is the output from serial port.

`[ 786.778538] reboot: Restarting system Orion2 CPU = High

Checking DATA BU Checking ADDRESS BUS

Checking hardware info ... === Strap status : 0x00003830 === === H/W boardId : 0x1f === === boardId : 0x14 === === micon_support: on === OK. === BUFFALO TS-HTGL R5 U-Boot. === LOADER ** BUFFALO BOARD: BUFFALO_BOARD_TS_HTGL_R5 LE (CFG_ENV_ADDR=fffff000)

U-Boot 1.1.1 (Oct 19 2008 - 21:02:54) Marvell version: 1.12.1 - TINY Buffalo Version: 1.19-1.00

DRAM CS[0] base 0x00000000 size 128MB DRAM Total size 128MB [256kB@fffc0000] Flash: 256 kB Addresses 20M - 0M are saved for the U-Boot usage. Mem malloc Initialization (20M - 16M): Done

Soc: 88F5281 D0 CPU: ARM926 (Rev 0) running @ 500Mhz Orion 2 streaming disabled VFP not initialized SysClock = 250Mhz , TClock = 166Mhz

USB 0: host mode PCI 0: PCI Express Root Complex Interface PCI 1: PCI-X, speed = 133000000 PCI_CONFIG dev7 count=0 data:00004000 PCI_CONFIG dev7 count=1 data:604211ab PCI_CONFIG dev7 count=2 data:604211ab PCI_CONFIG dev7 count=3 data:01000002 PCI_CONFIG dev7 count=4 data:01000002 PCI_CONFIG dev7 count=5 data:02b00000 PCI_CONFIG dev7 count=6 data:fff00004 PCI_CONFIG dev7 count=7 data:ffffff01 PCI_CONFIG dev7 count=8 data:00000000 PCI_CONFIG dev7 count=9 data:00000000 PCI_CONFIG dev7 count=10 data:00000000 PCI_CONFIG dev7 count=11 data:00000000 PCI_CONFIG dev7 count=12 data:00004000 PCI_CONFIG dev7 count=13 data:00004008 CPU: Write allocate Disabled Net: egiga0 [PRIME] Using 88E1118 phy

Marvell Serial ATA Adapter Found adapter at bus 1, device 7 ... Scanning channels Device 0: OK Model: WDC WD5000AAKS-22A7B0 Firm: 01.03B01 Ser#: WD-WCASY1010109 Type: Hard Disk Supports 48-bit addressing Capacity: 476940.0 MB = 465.7 GB (976773168 x 512) Device 1: OK Model: WDC WD5000AAKS-22A7B0 Firm: 01.03B01 Ser#: WD-WCASY1448619 Type: Hard Disk Supports 48-bit addressing Capacity: 476940.0 MB = 465.7 GB (976773168 x 512) Device 2: OK Model: WDC WD5000AAKS-22A7B0 Firm: 01.03B01 Ser#: WD-WCASY1000940 Type: Hard Disk Supports 48-bit addressing Capacity: 476940.0 MB = 465.7 GB (976773168 x 512) Device 3: OK Model: WDC WD5000AAKS-22A7B0 Firm: 01.03B01 Ser#: WD-WCASY0815609 Type: Hard Disk Supports 48-bit addressing Capacity: 476940.0 MB = 465.7 GB (976773168 x 512)

Using device ide0, partition 1

Loading from block device ide device 0, partition 1: Name: hda1 Type: U-Boot File:/uImage.buffalo

2063608 bytes read Image Name: kernel 4.12.13 Created: 2020-11-03 0:19:52 UTC Image Type: ARM Linux Kernel Image (uncompressed) Data Size: 2063544 Bytes = 2 MB Load Address: 00008000 Entry Point: 00008000 Using device ide0, partition 1

Loading from block device ide device 0, partition 1: Name: hda1 Type: U-Boot File:/initrd.buffalo

6447063 bytes read Image Name: ��F:#�dYXX�$c7�=H28`"�f���G3�<l
Created: 1970-01-03 21:10:21 UTC Image Type: Unknown Architecture Unknown OS Unknown Image (unknown compression) Data Size: 199605 Bytes = 183.8 MB Load Address: 752f7c46 Entry Point: 0f5b1a0c serch_boot_drv (250)>Bad majic No. 0 Using device ide1, partition 1 oading from block device ide device 1, partition 1: Name: hdb1 Type: U-Boot File:/initrd.buffalo

6447063 bytes read Image Name: ��F:#�dYXX�$c7�=H28`"�f���G3�<l
Created: 1970-01-03 21:10:21 UTC Image Type: Unknown Architecture Unknown OS Unknown Image (unknown compression) Data Size: 192699605 Bytes = 183.8 MB Load Address: 752f7c46 Entry Point: 0f5b1a0c serch_boot_drv (250)>Bad majic No. 1 Using device ide2, partition 1

Loading from block device ide device 2, partition 1: Name: hdc1 Type: U-Boot File:/uImage.buffalo

2063608 bytes read Image Name: kernel 4.12.13 Created: 2020-11-03 0:19:52 UTC Image Type: ARM Linux Kernel Image (uncompressed) Data Size: 2063544 Bytes = 2 MB Load Address: 00008000 Entry Point: 00008000 Using device ide2, partition 1

Loading from block device ide device 2, partition 1: Name: hdc1 Type: U-Boot File:/initrd.buffalo

6447063 bytes read Image Name: ��F:#�dYXX�$c7�=H28`"�f���G3�<l
Created: 1970-01-03 21:10:21 UTC Image Type: Unknown Architecture Unknown OS Unknown Image (unknown compression) Data Size: 192699605 Bytes = 183.8 MB Load Address: 752f7c46 Entry Point: 0f5b1a0c serch_boot_drv (250)>Bad majic No. 2 Using device ide3, partition 1

Loading from block device ide device 3, partition 1: Name: hdd1 Type: U-Boot File:/uImage.buffalo

2063608 bytes read Image Name: kernel 4.12.13 Created: 2020-11-03 0:19:52 UTC Image Type: ARM Linux Kernel Image (uncompressed) Data Size: 2063544 Bytes = 2 MB Load Address: 00008000 Entry Point: 00008000 Using device ide3, partition 1 `

1000001101000 commented 4 years ago

Data Size: 192699605 Bytes = 183.8 MB

That is more than 7mb

DavidDah commented 4 years ago

Now I'm bit confused, how does it update-initramfs creates 183 MB file uncompressed? I checked size of initrd.buffalo before reboot and size was 4871276 which is 4.8MB.

1000001101000 commented 4 years ago

Judging by the bogus size, junk for name/etc and bad checksum it seems the initrd.buffalo is corrupt somehow.

I’ve not seen that but I haven't looked that closely at the console at that phase of boot to be honest.

The bytes read looks too big for 4.8mb, i’m guessing that’s a bad From from when the xz settings were wring or something like that

DavidDah commented 4 years ago

I will do now the following, connect to serial port and start logging.

  1. Make clean install of debian
  2. run update-initramfs and reboot
  3. change compression algo to xz and run update-initramfs again and reboot If that works then i will try installing ntfs-3g again and reboot after installation. I suspect that I will run in bootloop again and hopefully be able to get some meaningful output on serial port.
1000001101000 commented 4 years ago

Sounds like the right approach.

Sorry I havent fixed the micro-evtd thing yet

DavidDah commented 4 years ago

updating initramfs after clean install worked, updating initramfs with XZ compression worked, apt-get install ntfs-3g invoked update-initramfs during package installation and after reboot system was crashed. HOWEVER after reboot, TS started normally. I didn't change anything i just rebooted it again (reset command from Marvel shell) and Debian started normally. I attached log from serial port session, maybe you can see something in it which i missed. screenlog.txt

1000001101000 commented 4 years ago

Just looking at the error and matching it to the corresponding kernel code I can see it has something to do with starting up the RAID:

[   29.030031] ------------[ cut here ]------------
 29913/655360 fi[   29.035657] WARNING: CPU: 0 PID: 118 at drivers/md/md.c:2278 set_in_sync+0x34/0xf8 [md_mod]
les, 247317/2619[   29.045220] Modules linked in: raid456 libcrc32c crc32c_generic async_raid6_recov async_memcpy async_pq async_xor $
136 blocks
[   29.065091] CPU: 0 PID: 118 Comm: md1_raid1 Not tainted 4.12.13 #3
[   29.072245] Hardware name: Buffalo Terastation Pro II/Live
static bool set_in_sync(struct mddev *mddev)
{
        WARN_ON_ONCE(!spin_is_locked(&mddev->lock));
        if (!mddev->in_sync) {
                mddev->sync_checkers++;
                spin_unlock(&mddev->lock);
                percpu_ref_switch_to_atomic_sync(&mddev->writes_pending);
                spin_lock(&mddev->lock);
                if (!mddev->in_sync &&
                    percpu_ref_is_zero(&mddev->writes_pending)) {
                        mddev->in_sync = 1;
                        /*
                         * Ensure ->in_sync is visible before we clear
                         * ->sync_checkers.
                         */
                        smp_mb();
                        set_bit(MD_SB_CHANGE_CLEAN, &mddev->sb_flags);
                        sysfs_notify_dirent_safe(mddev->sysfs_state);
                }
                if (--mddev->sync_checkers == 0)
                        percpu_ref_switch_to_percpu(&mddev->writes_pending);
        }
        if (mddev->safemode == 1)
                mddev->safemode = 0;
        return mddev->in_sync;
}

I don't really see what's going on but there are a few things you can try.

The first thing I'd recommend is checking the SMART data for each drive, I could envision a failing drive triggering this type of error.

Next, I'd actually try Debian Stretch and see if that is more stable.

It's possible this could be a bug with the RAID driver in the 4.12 kernel but I find that somewhat hard to believe. I'm not sure if I've set up RAID on this device but I've heard form others who have and haven't had this reported before. Still Stretch's 4.9 kernel is more stable overall and worth a try.

There reason the Terastation II/III are using the 4.12 kernel is because they made major changes to all the PCI drivers in 4.13 which broke PCI support on these devices. I spent some time looking into a fix but it's pretty far outside my current expertise.

DavidDah commented 4 years ago

I have checked drives with smartctl and they do not report any issues, however you might be onto something with raid bug in 4.12. I found bunch of articles regarding bug in raid such is this https://archlinuxarm.org/forum/viewtopic.php?f=53&t=11868

Also i have tried restarting TS several times from shell and it booted normally, however when i powered it off and started again it failed to boot. After i pressed power on button for second time it started normally.

root@debian:~# smartctl -H /dev/sda smartctl 6.6 2017-11-05 r4594 [armv5tel-linux-4.12.13] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED

root@debian:~# smartctl -H /dev/sdb smartctl 6.6 2017-11-05 r4594 [armv5tel-linux-4.12.13] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED

root@debian:~# smartctl -H /dev/sdc smartctl 6.6 2017-11-05 r4594 [armv5tel-linux-4.12.13] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED

root@debian:~# smartctl -H /dev/sdd smartctl 6.6 2017-11-05 r4594 [armv5tel-linux-4.12.13] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED

1000001101000 commented 4 years ago

Good job on the research on that one.

I'll see if I can hunt down that patch and add it to the 4.12 kernel in the repo and then generate installers with the updated kernel and the micro-evtd fix. It might be a few days but I've set the project up to make such changes relatively painless.

It's been a while since I looked into the PCI issue limiting us to 4.12, maybe I should see if anyone has done anything about it since I last looked. It has something to do with how non-DTB ARM devices initialize PCI. There were a ton of similar problems with network drivers for those devices around the same time. Part of the problem is there are only a few of these Orion5x/MV78100 devices with PCI and only a subset of those were supported by Debian (Sadly, I came to the party pretty late). It sounds like the other PCI devices were losing support for other reasons so I'm not sure if anyone has really looked into it.

1000001101000 commented 4 years ago

I've applied the patch, built a kernel, and tested that it boots on my devices and posted it to the repo.

you should be able to get it with: apt-get update; apt-get upgrade

I'll work on the installer part later, particularly if you find it solves your issue.

DavidDah commented 4 years ago

Wow this was FAST! I really appreciate the effort which you are putting in this project. 👍
I can confirm that in both cases (reboot and power off) TS is booting normally now. I will try few more things like intentionally cutting power off and check what will happen. BTW do you know name of the package which Buffalo was using on stock firmware for UPS communication to power on TS once when power is restored? It seems that i can now finish my installation guide for TS PRO II and move to next TS to switch it to Debian :)

Linux debian 4.12.13 #4 Tue Nov 3 14:18:55 CST 2020 armv5tel __ .
_ /___ / _// |__ / ||| __ _
| |/ _ \ _ \ ____ \ __ \ \ |/ \ / \ | |\ /| | \// _ \/ | | / | | | ( <_> ) | \ |_| \ >| (__ /___ /|| (__ /| ||__/|_| / \/ \/ \/ \/ \/ Terastation Pro II TS-HTGL/R5

1000001101000 commented 4 years ago

I have never actually used a UPS.... unless you count some bad experiences at work a decade ago.

I believe I saw on a forum that Buffalo that they switched UPS software between models. It sounded like they were all based on open source projects.

DavidDah commented 4 years ago

it seems that they switch to nut https://networkupstools.org/ in latter versions of TS, i just installed it trough apt-get and managed to configure it for my Eaton 5E ups without too much hassle.

Here are nut rules from original buffalo firmware if someone needs them, i didn't use them at all since nut default configuration almost worked for me. nut_rules.txt

1000001101000 commented 4 years ago

nice.

Overall I haven't found much value in trying to replicate how Buffalo did things since a lot has changed over the years but if you have something that works under the stock firmware it's a good place to start.

DavidDah commented 4 years ago

I think that original buffalo firmware's are outdated by all standards, buffalo was very conservative company since they entered the market with NAS and wireless products, and when i say conservative I mean that they always implemented the things which were maybe little bit outdated but proven and tested. They were never in the rush to add some new untested features into their products, it's just different philosophy which i tend to like very much because stability should always be on the first place.

1000001101000 commented 4 years ago

Yeah, I definitely appreciate the different constraints they work under trying to deliver an appliance style device rather than a generic server. It was a very different time with respect to ARM support in the linux kernel back then too. They do a ton of stuff in kernel that I do with userspace tools and device-trees now but that was the only option back then.

For a long time I stayed away from kernel code entirely since the devices I was using all ran just find on Debian's default kernel with only minor drawbacks. Then I started working on the Terastation II/III and Intel Devices which all require some kernel work, suddenly I'm really glad I took those C classes 20 years ago.

DavidDah commented 4 years ago

I would still like to know how did they handle USP recovery function on this old terastations, it has to do something with In most server BIOSes you can set the system to restore power state from before power loss. This means, that if the server is powered down and power is lost, it will remain powered down after you plug it in. On the other hand, if it was powered on, when the power was lost, it will power up when AC is restored. This option was working with their stock firmware, but I assume that they had to pass some parameter to u-boot or to Marvell loader.

1000001101000 commented 4 years ago

I think that is one of the reasons they use that microcontroller for power stuff, or why they kept using it so long.

I've documented most of the things I've figured out about the protocol from some of the GPL code that uses it, there's also a debug option on the tool the stock firmware uses to talk to it. There are several commands related to UPS: https://buffalonas.miraheze.org/wiki/Terastation_Microcontroller_Interface

You can experiment with sending those commands using micro-evtd or the python library: https://github.com/1000001101000/Python_buffalo_libmicon https://github.com/1000001101000/micro-evtd

DavidDah commented 4 years ago

Thank you very much, I will go trough that very thoroughly, in the meantime i found in binary file /usr/local/sbin/apcupsd strings which points that they used procfs to set recovery flag on. So if I can find how to control that flag with micro-evtd it would be awesome! Entered /etc/melco/ups ups_recover= =on %s> Recover flag = on %s> Recover flag = off /proc/buffalo/ups/ups_recover usb_on %s> Verify succeeded Set ups recovery flag failed... %s> Leaving(with return code = %d) %s> Cant open config No need to set ups recovery flag... Set ups recovery flag success. System automatically boot up after linefail recovering. %s> Verify failed SetupUpsRecoverFlagAuto IsUseUpsRecover VerifyUpsRecoverFlagUsb

1000001101000 commented 4 years ago

I took a quick look at the source that populates that /proc entry. I see code in there for microcontroller stuff and for passing "magic" values to the rtc.

The only device I've previously confirmed sending "magic" rtc values is the LS-QL. I created a script to handle sending shutdown commands to it: https://github.com/1000001101000/Debian_on_Buffalo/blob/master/Tools/rtc_restart.sh

The code mentions a DS1339 RTC which I believe was only used by this device, so I'm guessing that's part of it.

If you download the kernel source for that dev the relevant code appears to be in:

arch/arm/mach-mv88fxx81/Board/buffalo/BuffaloGpio.h
arch/arm/mach-mv88fxx81/Board/buffalo/BuffaloGpio.c
buffalo/drivers/buffalo_upsdrv.c
buffalo/drivers/BuffaloGpio.h

I assume the purpose of all of this is to be able to instruct the device to power on automatically after power failure (or not)..... Though my lack of UPS experience could have me thinking about it wrong.

If that's the goal I'd think we'd look for a combination of microcontroller commands and rtc values that causes the desired behavior and then make a script to handle it that the ups daemon would trigger at the right times.

That's basically how the shutdown/restart handlers work for these devices currently, with a mix of approaches for different devices.

1000001101000 commented 4 years ago

...though for this models some of that is handled by kernel code someone submitted a long time abo rather than my scripts: https://github.com/torvalds/linux/blob/master/arch/arm/mach-orion5x/terastation_pro2-setup.c

Doesn't look like they created any UPS features, but extending that might be an option too, though I still prefer userspace where possible.

DavidDah commented 4 years ago

I take quick look at kernel code, it doesn't say anything about ups functionality, but if we take look at UPS shutdown/recovery process it should be rather simple: NUT or APCUPS deamon get from UPS over USB or serial notification that AC is off.

Nov 4 18:53:11 debian upsmon[663]: UPS eaton5e@localhost on battery

After that nut initiates shutdown according to the ups.conf (usually 3 min delay)

SHUTDOWNCMD "/sbin/shutdown -P now" POWERDOWNFLAG /etc/killpower

NOTIFYFLAG ONBATT SYSLOG+WALL+EXEC NOTIFYFLAG ONLINE SYSLOG+WALL+EXEC NOTIFYCMD "/etc/nut/notifycmd"

UPS disables AC output on outlets when it's battery reaches critical output and remains that way until AC is restored. On x86 compatible PC there is usually option in BIOS RESTORE ON AC LOSS which means that as soon as AC is back PC will start. For TS it should work on similar way, I tried to do root@debian:/var/log# micro-evtd -s 0x028 0 0x028 should be shutdown_ups_recover

After that i initiated poweroff with shutdown -P NOW and then disconnected AC power from TS and reconnect it back after couple of minutes. Unfortunately it didn't start automatically, so I assume that either 0x028 is wrong flag or I just don't understand how this process works on TS.

BTW I assume that -s flag on micro-evtd should be send, is there get flag to read status of micro controler?

1000001101000 commented 4 years ago

Looking at my ts2pro it clearly doesn't have a DS1339 that code might be from even older devices though it describes more or less how the LSQL. I've really only looked at this stuff in the context of shutdown vs restart though.

For the LSQL the board always powers on after the power is cut, it then starts u-boot which reads the magic numbers from the RTC and depending on the value either continues the boot or waits for the power button to be pressed. You would see it in the serial console if it was doing that.

For this device the power button seems to be wired directly to the micocontroller. For the micro controller to check "magic" values they would need to be stored somewhere that persists through a power disconnect. It could be that it does this by reading the RTC or it could have some storage internal to itself or something.

The syntax of that micro-evtd command is incorrect, to send the signal you're trying it would be: micro-evtd -s 0028

That won't actually work since this device doesn't seem to support that message. if you run micon_dump.py from the Python library it will display all the readable values for the device and what their current value is.

DavidDah commented 4 years ago

Thank You for explaining me how this process work, so in order to find exact value I just need to dump all values with micon_dump.py and test them and see which value will work.

I'm trying to get micon_dump working but it requires serial module and i'm not sure which one...

root@debian:~# python3 micon_dump.py Traceback (most recent call last): File "micon_dump.py", line 3, in import libmicon File "/root/libmicon.py", line 6, in from serial import Serial ImportError: cannot import name 'Serial' from 'serial' (/usr/local/lib/python3.7/dist-packages/serial/init.py) root@debian:~# pip3 show pyserial Name: pyserial Version: 3.4 Summary: Python Serial Port Extension Home-page: https://github.com/pyserial/pyserial Author: Chris Liechti Author-email: cliechti@gmx.net License: BSD Location: /usr/lib/python3/dist-packages Requires: Required-by: root@debian:~# pip3 show serial Name: serial Version: 0.0.97 Summary: A framework for serializing/deserializing JSON/YAML/XML into python class instances and vice versa Home-page: https://bitbucket.com/davebelais/serial.git Author: David Belais Author-email: david@belais.me License: MIT Location: /usr/local/lib/python3.7/dist-packages Requires: pyyaml, iso8601, future Required-by: root@debian:~# apt-get install python3-serial Reading package lists... Done Building dependency tree
Reading state information... Done python3-serial is already the newest version (3.4-4). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.