Doodle3D USB storage / samba-server / other things

olijf commented 8 years ago

Last week I did some experiments with external USB storage.

USB Storage

I added kmod-scsi-core kmod-usb-core kmod-usb-storage kmod-usb-storage-extras block-mount kmod-fs-vfat kmod-nls-* packages to the standard Wifi-Box image. Unfortunately the complete samba36-server package necessary for network shares is far too big to fit on the Wifi-box internals.

With these packages it is fairly easy to add external storage + automount with block-mount Example config (import with uci import):

package fstab

config global
        option anon_swap '0'
        option anon_mount '0'
        option auto_swap '1'
        option auto_mount '1'
        option delay_root '5'
        option check_fs '0'

config mount
        option target '/mnt/sda1'
        option enabled '1'
        option options 'rw'
        option enabled_fsck '0'
        option device '/dev/sda1'
        option fstype 'vfat'

So now I have USB storage. I tried printing a file from USB with p3d -f <gcode-file>. Unfortunately print3d is designed to allocate all memory necessary to hold the file before starting to print, the wifibox has 32MiB of RAM available so this makes it impossible to hold larger files.

Extra storage = extra swap?

To add a swapfile to the SD card you just dd if=/dev/zero of=/<sd card swapfile location> bs=1024 count=<size of your swap in bytes> then enable it in fstab:

config swap
        option device '/mnt/sda1/.swapfile'
        option enabled  '1'

Now print3d should have enough space to hold a larger file. Calling it with these settings just returns null without much info. So this does not work right away. It looks like print3d has to be modified to allow for reading the file in chunks.

OverlayFS to add extra storage to standard folders.

I mounted the SD card to the mount -t overlayfs -o lowerdir=/root/sketches,upperdir=/mnt/sda1 overlayfs /root/sketches folder to allow for some extra space for doodles. This seems to work quite decently with easy access using the file manager. Once the USB storage is removed the Wifi-Box just continues to function properly.

Some thoughts on vfat

It seems that vfat is very prone to errors. If an SD card is not correctly unmounted/ejected it corrupts the filesystem fairly easy. Before the next mount it is recommended to run fsck once to fix any errors sudden disconnects/power losses etc might have caused.

Extroot using the USB Storage

It looks like it is possible to use vfat as external storage although this seems to mess with certain file permissions not present in this file system. Also when I tried to run samba36-server installed on the USB it did give some issues with configs in different locations than expected.

Which way to go now?

It looks as if it is not that much trouble to add an extra file manager which you can use to upload your GCODE files to the external storage and create a small print button. The only thing holding this back seems to be the print3d driver not supporting reading a file in chunks. For this to work we must modify the logic of the gcodeAppendFile function in https://github.com/Doodle3D/print3d/blob/0280dd5c36eec7bf5a7cf3872c1b09df388a20fc/src/server/CommandHandler.cpp#L145 could @woutgg give an indication of how much work this would be/tips on how to do this?

Adding USB storage would also eliminate some trouble we have had in the past with network connectivity issues whilst uploading to the Wifi-box. It seems that with the addition of the Cura Wifi-box plugin that this would make certain things much easier.

woutgg commented 8 years ago

Nice work, interesting to see the various options on using external storage. As for changing how the print server deals with printing from files, I think we would need these changes:

Change hnd_gcodeAppendFile() to only tell the driver which file to print.
Have the gcode buffer (since this is not driver-specific) open the file and read lines as needed, this can for instance be done using an iterator, like described here. Memory mapping is an option but I'm not sure that would improve performance.
The buffer class would be extended with an update function which reads lines and appends them using append(). This could be done with the same strategy as in the client; i.e., completely fill the buffer, then wait until below 75% and repeat.
One thing we'd still need to solve is how to report correct progress. Is line counting feasible? Otherwise we could somehow use the file size to get an estimate (e.g. to extrapolate current size/lines ratio), or allow passing this information along like the client currently already does.

Reading the file while it is still being written to should work fine, but it would be good to test this if we want to support that.

Regarding an estimation of time required, I'd say 1 day. So after doubling that, 2 days.

peteruithoven commented 8 years ago

Would it be possible to extend the available RAM memory? So Print3D could hold all the data in it's existing buffer?

woutgg commented 8 years ago

That is what creating a swap file effectively does. And I would indeed like to investigate why the server only returns null, it should at the very least be more informative.

(For anyone interested in some background on virtual vs physical memory and the OOM killer we have already dealt with: I just ran into an interesting article on this subject.)

Doodle3D / WiFi-Box