search

lukehutch / usb-copier

USB drive copier example project for Raspberry Pi with the Adafruit 128x64 bonnet
MIT License
18 stars 7 forks source link

Could not resolve dependencies - adafruit-oled-bonnet-toolkit #1

Open 2wenty2wo opened 3 years ago

2wenty2wo commented 3 years ago

I'm not sure how to fix but it failed to compile with the following error:

-----------------------< usb-copier:usb-copier >------------------------
Building USB Copier 0.0.2
--------------------------------[ jar ]---------------------------------
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.4/maven-install-plugin-2.4.pom
Progress (1): 2.7/6.4 kB
Progress (1): 5.5/6.4 kB
Progress (1): 6.4 kB    

Downloaded from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-install-plugin/2.4/maven-install-plugin-2.4.pom (6.4 kB at 4.8 kB/s)
The POM for adafruit-oled-bonnet-toolkit:adafruit-oled-bonnet-toolkit:jar:0.0.2 is missing, no dependency information available
------------------------------------------------------------------------
BUILD FAILURE
------------------------------------------------------------------------
Total time:  3.043 s
Finished at: 2021-02-23T22:41:18+11:00
------------------------------------------------------------------------
Failed to execute goal on project usb-copier: Could not resolve dependencies for project usb-copier:usb-copier:jar:0.0.2: Failure to find adafruit-oled-bonnet-toolkit:adafruit-oled-bonnet-toolkit:jar:0.0.2 in https://oss.sonatype.org/content/groups/public was cached in the local repository, resolution will not be reattempted until the update interval of oss-snapshots-repo has elapsed or updates are forced -> [Help 1]

To see the full stack trace of the errors, re-run Maven with the -e switch.
Re-run Maven using the -X switch to enable full debug logging.

For more information about the errors and possible solutions, please read the following articles:
[Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
lukehutch commented 3 years ago

Oh, sorry, you need to download this dependency, and mvn install it:

https://github.com/lukehutch/Adafruit-OLED-Bonnet-Toolkit

2wenty2wo commented 3 years ago

Thank you for that, I really appreciate it, I installed the dependency which allowed NetBeans to make the jar file. 👍

Steps I've taken:

Not quite sure what to do next?

Thanks 😄

lukehutch commented 3 years ago

Oh, sorry! I never expected anyone else to actually try to run this code, and you're the first person who has tried :-) (Is this Declan? GitHub doesn't show me your real name.)

I ran into the same issue, and reported it months ago, but didn't get a response: https://github.com/Pi4J/pi4j-v2/issues/39

So instead I resorted to just manually copying out the native library that I needed from that jar.

I forgot that I made some notes on getting this all working... these may not be 100% correct, but please work through these and let me know if you run into any problems. Then I'll put the instructions on the README.md page.

2wenty2wo commented 3 years ago

IMG_4400

Thanks again Luke, with all the information you provided I've got this to work. 😄

No I'm not Declan, my names Shaun, I don't know you personally, I was just looking for a USB duplicator as I'm an arcade technician and I'm constantly needing recovery USBs for games and operating systems etc, having a dedicated USB duplicator is a very helpful tool to have and I won't need to tie a computer up using Clonezilla. Searching around the internet I found your GitHub page. I already had the Raspberry Pi Zero but ordered the USB Hub and OLED bonnet as that's what your code was based on. Only issue is I have zero Java experience. 😮

These are the steps I took:

Build Machine (MacOS):

Raspberry Pi Zero:

In your TODO file, what did you mean by "Need to unmount after copying has finished, not just sync (otherwise drive dirty bit is set)"? Is there an option to unmount the disks in the UI?

I'm currently testing it to see what it can copy. It copied a normal USB with some random files quickly which is nice. I'm currently attempting to clone a Windows recovery USB to see if it will still be bootable, it's been running for some time now with no progress on the bar, but the USB LED is flashing so something is happening.

I really need to learn Java because I'd love to implement a shutdown option in the menu and a couple of other ideas. This software you've written is awesome, I have no idea how I'm the first person to attempt to try this code!

lukehutch commented 3 years ago

Wow, I'm impressed you got all that working (hardware + software) from the very scant information I provided!

This was only really a prototype, and you may run into some bugs and limitations. In particular it doesn't copy the entire drive, only the largest partition (mostly because drives can be different sizes, so copying the partition table over could easily result in a drive that is not readable, since the partition table may indicate the drive is a different size than it is). Therefore a recovery disk probably won't copy well. If you are always copying between drives that are identical in geometry and size, then you could change the copy command in the code to just dd from the base block device for the whole source drive over to the base block device for the whole destination drive (e.g. /dev/sda to /dev/sdb).

I didn't realize there was a TODO file in the repo -- everything to do with udisks2 is obsolete. I had too many problems with it, so I switched to udevil instead. I'll have to fix that TODO file (or just remove it).

I'm glad you still got it working though. I know Java is not a language that most Raspberry Pi hackers use...

One thing you should be aware of is that if you use a USB hub shield on the Raspberry Pi Zero, the screen may freeze and the device may reboot when you plug a drive in. This is known as the "inrush problem". Raspberry Pi Zero skimps on power management components, so can't handle the initial power draw of many USB thumb drives when they are plugged in, without the core voltage rails dropping so low that a reboot is triggered. I actually had to switch to a Raspberry Pi 4B for the latest prototype.

Also the pi4j dependency may have some glitches on Raspberry Pi Zero, with GPIO pins being reordered or something, e.g. the dpad or buttons may not operate as expected. Do all those buttons work for you?

One more tip, you can add the java launch command to /etc/rc.local if you want this to run on boot -- but booting takes about 30-40 seconds, and the screen won't show anything before then.

2wenty2wo commented 3 years ago

Firstly, thanks so much for replying to me so quickly and guiding me 😄 , it was all your help! I'll be honest, I was pretty damn excited when your menu finally loaded up on the OLED display! 😛

I believe you're currently using rsync to copy the files over, is that correct? I do want to use dd to duplicate drives so it's a 1:1 copy (I would never use this just to copy some random files from one drive to another, I would only ever use this to copy 'restore/install' USB's), I'm just not sure how to implement it into your code. 😟 You're not wrong, Java isn't a language that's very common in the Raspberry Pi world, I mostly play around with Python.

I know with Clonezilla, you can make a 1:1 copy of a USB drive as long as the destination drive is equal to or greater in size. Is this something I can implement into your 'usb-copier' program? I understand that this is just a prototype program to demonstrate your Adafruit OLED Bonnet Java Toolkit but I really do believe this has real-world uses (especially in my field). I'd seriously pay for this software.

I haven't had any issues with the USB hub yet (touch wood), but I will keep an eye on it. I had it running all last night trying to copy a restore USB that is basically a Windows embedded install, It didn't restart but in saying that the progress bar didn't move at all (8 hours or so), as you said it's only copying the largest partition so it probably wouldn't have worked regardless if it finished or not.

Regarding the pi4j dependency, I haven't encountered an issue with the d-pad and a/b buttons, everything seems to work as expected which is nice.

I did add the java launch command to /etc/rc.local and you're right during boot the screen is blank until the OS has loaded and the command has run, this is no problems for me, I just wait until it says "please wait" and asks for language selection.

I'm not sure if you can or have time to help me but here are some things I'm looking for:

Some nice to haves:

Lastly, I just wanted to thank you again for sharing this code free of charge and helping me along the way. Do you have a donation option, would happily send you something.

P.S. I saw your profile on Linkedin and I do feel very underqualified to talk to you regarding computers so thank you for your time and help haha. 👍

Cheers! Shaun

lukehutch commented 3 years ago

I believe you're currently using rsync to copy the files over, is that correct?

Correct.

I do want to use dd to duplicate drives so it's a 1:1 copy (I would never use this just to copy some random files from one drive to another, I would only ever use this to copy 'restore/install' USB's), I'm just not sure how to implement it into your code.

Here's the current rsync command:

https://github.com/lukehutch/usb-copier/blob/master/src/main/java/screen/DoCopyScreen.java#L132

Just replace the varargs parameter at the end (which is turned into an array of Strings by Java) into the command that you want. For example

                    "rsync", "-rIlptv", "--info=progress2",
                    // End source dir in "/" to copy contents of dir, not dir itself
                    selectedDrive.mountPoint + "/", //
                    destDrive.mountPoint

into the following to use dd on the whole drive:

                    "dd", "if=" + selectedDrive.rawDevice, "of=" + destDrive.rawDevice, "bs=8192",
                    "status=progress", "oflag=direct"

See this for rawDevice

https://github.com/lukehutch/usb-copier/blob/master/src/main/java/util/DriveInfo.java#L53

Also if you want the progress bar to update during the dd copy, you need to replace the progress line parsing code (which is designed to parse for rsync process output) into a form that can parse dd output with the status=progress switch. An example of this is here, from the low-level disk wipe command:

https://github.com/lukehutch/usb-copier/blob/master/src/main/java/screen/DoWipeScreen.java#L76

Unfortunately this is where it gets a bit more tricky, because this USB copier code uses totally custom process management code so that the whole UI is asynchronous and non-blocking, while being able to launch any number of processes in parallel, with readers consuming input from the output of commandline processes, and allowing for tasks to be canceled, etc. In the most recent link above you can see that Exec.execConsumingLines is called. The first arg is null (the stdout consumer) and the second arg is a lambda that is called for every line of stderr output from the dd command. So you'd need to insert the dd stderr progress line consumer lambda, from the above link, at the following location in place of the rsync progress line consumer lambda:

https://github.com/lukehutch/usb-copier/blob/master/src/main/java/screen/DoCopyScreen.java#L106

but you'll need to insert a null before the lambda, since you want to consume stderr for dd progress, not stdout. You'll also need to modify the code, following the pattern shown, to set the progress bar appropriately: progressBar.setProgress(percentageInt, 105) etc.

I hope that makes sense... it's not a lot of work in all, but it's a bit complicated to understand what's going on, thanks to the (overengineered) async process architecture! Sorry about that...

Regarding the pi4j dependency, I haven't encountered an issue with the d-pad and a/b buttons, everything seems to work as expected which is nice.

Great, glad to hear it.

  • Linux/Windows file system support (arcade game manufacturers use both Linux and Windows for their cabinets).

Switching to dd will solve this.

However, I have used dd many times to try to write ISO files to USB drives for Linux live USB images -- and for reasons that I don't understand, it just doesn't work for some bootable ISO images and/or some USB sticks.

  • 1:1 sector-to-sector USB duplicator that supports MBR etc (bootable drives).

Also solved by dd

  • Option in the UI to safely unmount drives.

This is not explained anywhere, but already drives can be unplugged whenever there is no ongoing disk operation. I was careful to call sync when a copy/wipe operation has finished, and in fact I think I even went as far as to unmount (or unmount and remount) each destination drive.

  • Option in the UI to shutdown after copy completion.

Honestly you can just pull the power whenever you're done copying drives! It shouldn't hurt the OS. In the worst case, you might have a missing or corrupted system log entry. But this USB copier doesn't even write to the OS SD card.

Time remaining on copy. Time elapsed on copy.

I agree, this would be useful. It would require adding a text label UI element with the predicted time in it, and modifying the code that updates the progress line to also produce a time estimate, maybe only after the first 10 seconds have elapsed.

Name of USB drive (if applicable).

This should be available in the DriveInfo.label field, e.g. selectedDrive.label or destDrive.label in the code mentioned above. I don't know how reliable this label info is though, and you'd need to add a UI element for this too.

You can see some layout code here (for the wipe operation):

https://github.com/lukehutch/usb-copier/blob/master/src/main/java/screen/DoWipeScreen.java#L104

Just add a new TextElement below the progressbar, in the same VLayout, and store the ref to it in a field. Then you can set the value of that status line from the progress parsing code.

Unfortunately I don't have a lot of bandwidth to work on these changes right now! But I hope this gives you some pointers to tinker with it.

Lastly, I just wanted to thank you again for sharing this code free of charge and helping me along the way. Do you have a donation option, would happily send you something.

Thanks, that's very generous! But you don't need to. I don't feel like I have done much to help at this point. I see being responsive and communicative about code I publish as just the responsibility I assume for putting my code out there. If you want to, you could find some aspiring kid programmer doing some great work, and send them a few bucks -- pass it on :-) The kids need encouragement more than "old" guys like me (44) :-)

P.S. I saw your profile on Linkedin and I do feel very underqualified to talk to you regarding computers so thank you for your time and help haha.

Oh, hah, you're too generous, but you're not underqualified, you are figuring this all out. Everybody with earnest questions or asking for help is worth an investment of time.

lukehutch commented 3 years ago

Hey @2wenty2wo -- somebody told me about this device. I think this is ideal for your particular usecase. You will save yourself a lot of time and effort by using something off-the-shelf like this!

https://www.amazon.com/StarTech-com-Standalone-Duplicator-Eraser-USBDUP12/dp/B00BOK3NQI

2wenty2wo commented 3 years ago

Hi @lukehutch, sorry for not replying to your previous comment, I was trying to see what I could do with your code/advice before saying anything haha. 😄

I had a go at adding the dd command into your code, it worked, I was able to get it to dd clone a USB stick but couldn't wrap my head around implementing the progress bar, so I had no idea if it was working or not. I waited until the morning and it did clone the stick. 👍

I thought I might as well try and make something myself but using something simpler like Python, currently, it doesn't work yet, I've been spending time trying to make a menu system that works with the OLED bonnet. I'm very much a novice so this will take me some time but it's been fun trying to learn this as I go. I made my first repository Rpi-USB-Cloner which has also been fun. I just want to make something basic for myself and if someone that is more skilled than me wants to make it better, I won't complain! 😆 I also linked to your repository as your project was what inspired me to do this.

Thank you for sending that link, that device looks very interesting and the price is reasonable for what it does. If my project falls apart, I'll definitely grab a turn-key USB duplicator from Amazon!

Thanks again Luke, you're a champion.

lukehutch commented 3 years ago

@2wenty2wo they say that necessity is the mother of all invention -- but it is also the origin of all growth! Actually this was my very first RPi project ever, and I learned a ton through building this. So if you value learning new stuff (rather than just wanting a turnkey solution), it's actually a great thing that you're diving into building your own equivalent with Python.

It sounds like you found the Adafruit OLED Bonnet Python library, so that you're not starting at zero like I did in Java? There's no menu system in that library, but it will at least render text on the screen for you, so you should be able to hack some basic textual interaction code together from that starting point.

Anyway, wishing you best of luck for your project!

alvin-000 commented 4 months ago

Sorry to bring back up an old issue, but I ended up seeing the project and really liked the features described in this chain, so I implemented most of them as a fork (a new DD copy menu with progress menu described above and wipe of entire disk). However, I noticed there was not built in functionality to read the raw disk (ex. /dev/sda) size due to the use of df. I tried to implement similar code from other code I see from DriveInfo.java and having it reference in DiskMonitor.java and the DoCopy clone Java code for DD, but it did not end up working. Can I get a pointer on how you might do this?

Added in code to DriveInfo.java to run lsblk to get selected device's raw disk size:

    public void updateRealDiskSize() {
        Exec.then(Exec.execWithTaskOutput("lsblk", "-b", "-n", "-d", "-o", "SIZE", rawDriveDevice), taskOutput -> {
        if (taskOutput.exitCode != 0) {
            System.out.println("Bad exit code " + taskOutput.exitCode + " from lsblk: " + taskOutput.stderr);
        } else {
            try {
                realDiskSize = Integer.parseInt(taskOutput.stdout);
                System.out.println("Full Disk Size (lsblk): " + rawDriveDevice + ": "  + Integer.toString(realDiskSize));
            } catch (NoSuchElementException e) {
                e.printStackTrace();
            }
        }
        });
    }
lukehutch commented 4 months ago

@alvin-000 very cool that you worked on this!

What do you mean by "it did not end up working"? I'm not sure where to start advising you :-)

alvin-000 commented 4 months ago

My bad on that. When I got the progress bar working, I found out diskSize is actually partition size of volume, which is a bit problematic for my progress bar when you have a multi partition volume (like a cloning a Raspian SD card). Review of the original code shows you used df to get the info, which does not include full/raw disk size.

So I aimed to just add in a function to DiskInfo.java to get that info.

Initialize public variable:

    public volatile int realDiskSize = -5;

Function to get full/raw disk size via lsblk. I am kind of assuming I can just take the sdrout and make it into a global variable. 

    public void updateRealDiskSize() {
        Exec.then(Exec.execWithTaskOutput("lsblk", "-b", "-n", "-d", "-o", "SIZE", rawDriveDevice), taskOutput -> {
        if (taskOutput.exitCode != 0) {
            System.out.println("Bad exit code " + taskOutput.exitCode + " from lsblk: " + taskOutput.stderr);
        } else {
            try {
                realDiskSize = Integer.parseInt(taskOutput.stdout);
                System.out.println("Full Disk Size (lsblk): " + rawDriveDevice + ": "  + Integer.toString(realDiskSize));
            } catch (NoSuchElementException e) {
                e.printStackTrace();
            }
        }
        });
    }

I then added references to the new function above wherever I saw driveInfo.updateDriveSizeAsync() as that looked like another update disk size function (so like DiskMonitor.java and later in the DD copy Java script. 

    /** Drive is mounted and drive metadata has been read. */
    static void driveMounted(String partitionDevice, String mountPoint, String label) {
        DriveInfo driveInfo = getOrCreateDriveInfo(partitionDevice);

        driveInfo.mountPoint = mountPoint;
        if (label != null) {
            driveInfo.label = label;
        }
        driveInfo.isPluggedIn = true;
        driveInfo.isMounted = true;

        // Use device letter as port. TODO: get USB port from /proc
        driveInfo.port = driveInfo.rawDriveDevice.charAt(driveInfo.rawDriveDevice.length() - 1) - 'a';

        driveInfo.clearListing();

        driveInfo.diskSize = -1L;
        driveInfo.diskSpaceUsed = -1L;

        System.out.println("Drive mounted: " + driveInfo);

        **driveInfo.updateRealDiskSize();**

        // Call df to get drive sizes (updated asynchronously).
        // Calls DiskMonitor.drivesChanged() only if successful.
        driveInfo.updateDriveSizeAsync();
    }

Lastly, I modified the dd execution code to use the new variable for the progress bar and the former MSG for files:

                    long bytesProcessed = Long.parseLong(stderrLine.substring(0, spaceIdx));
                    int percent = (int) ((bytesProcessed * 100.0f) / **selectedDrive.realDiskSize** + 0.5f);
                    progressBar.setProgress(percent, 105);
                    repaint();
                }
            }, //
                    "dd", "if=" + selectedDrive.rawDriveDevice, "of=" + destDrive.rawDriveDevice, "bs=4096",
                    "status=progress", "oflag=direct");
        layout.add(new TextElement(Main.UI_FONT.newStyle(), new Str(Msg.COPYING, selectedDrive.port)),
                VAlign.CENTER);
        layout.addSpace(1, VAlign.CENTER);
        layout.add(new TextElement(Main.UI_FONT.newStyle(), new Str(Msg.DISK_SIZE, **selectedDrive.realDiskSize**)), VAlign.CENTER);
        layout.addSpace(1, VAlign.CENTER);

Everything compiles fine and when I check the nohup.out file I can see my lsblk command is run. However, I do not see the print out for success or the error message built into the function. Review of the selectedDrive.realDiskSize is MSG on the device, shows my value is -5 (the initialize variable). Any ideas what I might be doing wrong?

lukehutch commented 4 months ago

OK, so I'm assuming you mean that the callback code that reads from taskOutput.stdout is never actually called?

Try extracting just the Exec code and your lsblk call, and run it on your local Linux machine, so that you can step through it in the debugger and see where it's failing.

It has been so long since I have looked at this that that's the best suggestion I have right now!

alvin-000 commented 4 months ago

Thanks for the suggestion, I think I might have actually figured out the issue from debugging with just Execit as suggested. It looks like the lsblkoutput includes "hidden" quotes and my intention to convert the value into an integer early to have it ready as a number for the progress bar function cause it to error out (hence why I saw the Linux command run, but no stderr or stdout output). Looks like I just need to figure out how to do like a regex or quick number extractor function and hopefully, everything will just work.

image