Bulk import?

[sberney]

Hey, is there a way to bulk import files? I am using OneDrive instead of Google Drive, so I just need to point it at a folder on my local computer, but I am only seeing the option to import one file at a time, which will take too long.

By the way, this tool looks awesome. I've been looking for something suitable for quite some time, and it wasn't until I added the word "RPG" into my search parameters that I actually found this. I had been thinking I might find myself needing to create my own.

[sberney]

If there isn't support for bulk import, I might be interested in adding it. I am very proficient in C#, albeit with web infrastructure rather than desktop gui. I have done a bit of XAML, and quite bit of React and other web gui work. Though I don't yet fully understand exactly what the project is really doing since I haven't reviewed it all. 😄

[sberney]

I'm also wondering if you have a CI pipeline set up -- it would be nice to know whether the master branch is in a good state.

Relatedly, there doesn't appear to be any developer environment guide. When I download and build, I'm getting "Unable to find directory {omitted}\gs" when it's looking for the Ghostscript directory. I'll see if I can figure it out, but my immediate thought is -- are there extra environment steps I need to take to set this up, and is there a CI pipeline definition I can look at that contains any dependencies.

[DSPaul]

You can bulk import files by selecting multiple files in the windows explorer dialog by SHIFT clicking to select a range, CTRL clicking to select individual files, or CTRL-A to select all, you get the idea. There is however no way at the moment to select a folder or have it automatically search subfolders but that shouldn't be too hard to implement so I might look into that in the coming week.

As for the CI pipeline, there is none at the moment because I mostly made this for myself rather than production and I have no experience setting up CI pipelines. There is however a manual system that I follow: all commits are done to the dev branch which is merged into the master branch whenever there is a new release so the master should always be clean and reflect the latest release.

As for the Ghostscript directory, the problem there is that I don't think I am allowed to include the ghostscript files in the repo due to licencing about redistribution so I manually downloaded them from their website and put them in a /gs directory in the bin folder, where the COMPASS.exe is located with all the dll's. The reason behind this and an explanation which files you'll need is described here https://github.com/dlemstra/Magick.NET/blob/main/docs/Readme.md#ghostscript. The easier solution that I would recommend would be to simply install ghostscript and comment out this line MagickNET.SetGhostscriptDirectory(Path.Combine(AppDomain.CurrentDomain.BaseDirectory,"gs")); in MainViewModel.cs which should also work. The reason you have to deal with the /gs folder at all is because I didn't want user to have to install a second piece of software to use this one. The ghostscript files are included in the installer and written to a /gs folder during installation so users never have to think about it, but it does complicate the situation for devs. I wish there was a better solutions but given that Magick.NET simply tells their users to provide their own ghostscript files, I don't think there is one or else they would be including it themselves as well. If you have any ideas, I would be happy to hear them.

Lastly I'm am glad to see some interest in this project. I haven't looked much at it recently because it does most of the things I need it to do at this point but there a still plenty of things that can be done to improve and I have a long list of ideas so I will jump back in this week starting with importing entire folders and subfolders.

[sberney]

Thank you for the detailed response! Please let me know if you get the bulk import working. I think I will continue to look through the codebase.

[DSPaul]

I have added a first implementation of bulk import in the dev branch. It adds a new import option "from folder" that allows you to select a folder and it will import all files inside, including those in subfolders and their subfolders and so on, because I figured that if you didn't want to search subfolders, you can just open the folder and import all the files by shift clicking the old way. Assuming the ghostscript issue is fixed and you can build the project, I would encourage you to try it out to see if it works for you and if it does, I'll merge it into master and make a new release.

I might make searching subfolders an setting in the future, along with letting you filter/ignore file types to only chose pdf's for example if you were to have all kinds of random stuff in your folder but this should do for now.

[sberney]

Awesome, I can't wait to try it out. Thank you for doing it!

[sberney]

I hit an error while importing, it appears to be a corrupt pdf file (I was not aware I had any, it worked last I checked?). The import process failed and I had to hit the stop button on VS. It needs some error handling! I removed the corrupt file only to eventually hit another corrupt file (that ought not be but oh well).

It has to download the files from OneDrive so it's pretty slow going (they're not all cached locally). I'm not sure if that's related.

[DSPaul]

It has to download the files from OneDrive so it's pretty slow going (they're not all cached locally). I'm not sure if that's related.

I'm fairly certain that that is the cause of the problem. It probably tries to open the pdf file as soon as the file path exists without waiting for the download to complete, which would explain why they are corrupted. I'll definitely add some error checking so it will skip over corrupted/partially downloaded files and show a warning that those did not complete instead of crashing and I'll do some research if there is a way to wait for the pdf downloads to complete before trying to open them.

[DSPaul]

So I added some error checking on the dev branch. If a file is corrupt for whatever reason, I have chosen to still import it but without metadata or a thumbnail as those require opening the file but it will still show up in the library.

The only way I could think of to check whether or not a file is done downloading is check the file size, wait a little and check it again until there is no difference but that's a pretty hacky solution so I'll hold off on that for now. I'm also starting to doubt if that is even needed and causeing the problem because I was unable to recreate it. These are the steps I took:

• I uploaded some pdf's to Onedrive through the site so they definitely weren't cached locally.
• I then downloaded the Onedrive app which created a linked folder on my pc which I assume is your setup. All the files I uploaded were listed inside with a cloud icon, confirming that they weren't downloaded.
• I then tried importing the local onedrive folder in Compass and it worked just fine, although it took a lot longer, and at the end, I checked the files and they are now all marked with a solid checkmark, confirming they were downloaded in the process.

[sberney]

That appears so far to work on a subset of my files. I haven't had time for a proper test of it due to some Thanksgiving preparations. I will definitely have time to get this sorted out in a week or so, possibly sooner. Thanks for your efforts thus far!

[sberney]

I tried bulk importing my full library again, which contains a lot of images I don't really want and other miscellaneous stuff, but I hit this error, which halted the process:

[DSPaul]

Seems to be another corrupt file problem. I don't know why you keep having these. I added some error handling to the code in question so it skips rather than crashes. It could also be a problem with imagemagick so if you could share the file I can try see what's wrong with it.

[DSPaul]

As I was planning, I now also added a new dialog after you select a folder that will ask you which filetypes you would to import so you can filter out stuff you don't want.

[DSPaul]

I have published v0.8.0 which includes the new bulk import so I will close this issue, if you have any other problems or crashes feel free to open a new one.