Spatial file manager - Githubissues

probonopd commented 3 years ago

The current https://github.com/helloSystem/Filer/ implementation is not spatial.

Spatial Interfaces are explained by John Siracusa at https://arstechnica.com/gadgets/2003/04/finder/2/

It would be nice if we could have a spacial file manager.

probonopd commented 3 years ago

Also see https://www.asktog.com/columns/034OSX-FirstLook.html

probonopd commented 3 years ago

Possibly store the metadata for this using extended attributes?

ghost commented 3 years ago

Were you thinking that Filer should be extended to have a spatial mode or having an alternative to Filer that is spatial? I don't think there's any existing project in Qt (Gnome 2 had spatial mode at one time - we don't want that)

probonopd commented 3 years ago

Either way. One thing I don't like about Filer is that it still draws in some glib based components, and I was considering alternatives including DFilemanager but I could not get it to compile yet: https://github.com/helloSystem/Filer/issues/23

ghost commented 3 years ago

I have started extending Filer for spatial mode support - https://github.com/helloSystem/Filer/pull/35

I agree it would be nice not to have the glib components at all, but I think the current Filer is probably the best option still and for pragmatism it might be worth sticking with it and hope that eventually it can be refactored (well, libfm-qt)? I got DFilemanager to compile but there are more problems there.

ghost commented 3 years ago

Extended attributes looks viable - this is the next step in extending Filer. As a proof-of-concept from the command line:

HelloChris% setextattr user window.height 100 Documents
HelloChris% getextattr user window.height Documents 
Documents       100

See https://www.freebsd.org/cgi/man.cgi?setextattr

ghost commented 3 years ago

I had some spare time (I'm on 'holiday' this week :) ) so I went ahead with the extended attribute implementation for window position and size (modified the same PR).

This implements spatial folder browsing, so obviously there's an outstanding task for storing icon position as well but I think this is good to start with?

~We might want to hide the sidebar and navigation controls when in spatial mode to provide a similar experience to Haiku's tracker or classic Mac finder.~ OK, I had a bit more time and sorted this out too :)

It would also be nice if the 'maximise' window button behaviour would do the same as tracker in this case (resize to the minimum size to fit all items without scrollbars). That's a different topic though, I guess!

probonopd commented 3 years ago

Wow @moochris this is awesome! Thank you very much.

Let's think about a few more aspects of spatial navigation Maybe we should think about creating a spec for the metainformation needed to make a truly spatial file manager first, ideally in a way so that any file manager could implement the spec in a compatible way, e.g.,

Window size in pixels at normal (no-HiDPI) resolution
Window position on screen in a coordinate system
Display mode (icons view, list view, Miller columns,...) and a definition of fallbacks (e.g., if Miller columns were set but the file manager does not have them, then use list view)
Icon size for the window
How icons are positioned (automatic/by type/sorted/manual...) and sort order
Icon positions in a coordinate system for each icon
Window background wallpaper (only in icon view)
Custom icons for files and folders pasted by the user
Tags/colors of files
... As for storing the information, we should think about consider
Extended Attributes (advantage: no additional files cluttered around for storing this information)
One hidden file per directory (advantage: survives archiving and copy operations even over the network, etc.)
Possibly a combination of the two or something else.

We should investigate how other spatial file managers are doing this.

probonopd commented 3 years ago

More food for thought from reviewer John Siracusa and Classic Mac UX auhtority Bruce "Tog" Tognazzini:

For me, the possibility to open new windows for each folder is just one (and probably the most controversial - hence Apple made it optional in Mac OS X) aspect. Other aspects should also be considered, such as remembering for each location in which mode (list/icons,...) it is displayed, where each icon is (in icon view mode), how things are sorted (in list view mode).

ghost commented 3 years ago

Yep I think a spec would be a good idea.

I started on a kind of structure for the names of the extended attributes in the initial implementation, but it probably won't scale to include the other fields.

I had thought of a few similar to what you have listed, but also another nice thing might be to reuse Haiku's HVIF format to store icons in extended attributes? (I'm sure you're aware of this, but https://www.haiku-os.org/articles/2006-11-13_why_haiku_vector_icons_are_so_small/ link anyway :smile: )

And yes storage is definitely a consideration - e.g. it would be nice if Nextcloud (or Dropbox or whatever) could synchronise the metadata. I quite like the extended attributes (seem to work well in this first attempt) but I guess if they are transparent to the user, then some other file based mechanism like dot files would be preferable.

probonopd commented 3 years ago

I quite like the extended attributes (seem to work well in this first attempt) but I guess if they are transparent to the user, then some other file based mechanism like dot files would be preferable.

Let's have a look at where other systems are storing which kind of information; Haiku and the Mac are coming to mind immediately.

Speaking of Haiku:

Haiku Vector Icons

Yes, I know about them but since many applications don't have them I think we needd to support png and svg in any case.

ghost commented 3 years ago

An interesting post (albeit from a while ago now) on extended attributes:

https://www.lesbonscomptes.com/pages/extattrs.html

Hopefully some of those pitfalls have been solved in the meantime. I note that it mentions ROX using them, so worth adding that to the list of projects to investigate.

probonopd commented 3 years ago

Possibly we need to consider a combination of extended attributes (for performance) and files (for robustness) (I think this is what Haiku is doing). In any case, the spatial information should travel with the folder, so that e.g., if someone accesses the same folder over the network the spatial information should be available to that user, too.

probonopd commented 3 years ago

Need to keep in mind use cases like this one:

Here, whoever created that disk image

Set the sidebar to disabled for that window
Set a custom window wallpaper
Set the size and location of the icons

...and all of it traveled along inside the disk image.

ghost commented 3 years ago

Possibly we need to consider a combination of extended attributes (for performance) and files (for robustness) (I think this is what Haiku is doing). In any case, the spatial information should travel with the folder, so that e.g., if someone accesses the same folder over the network the spatial information should be available to that user, too.

I think Haiku only uses attributes for Tracker - it's these here: https://git.haiku-os.org/haiku/tree/src/kits/tracker/Attributes.h

e.g. _trk/windframe

But yes I agree these things should be portable. For Haiku, for example, they modified zip and unzip to preserve attributes. We should look into whether other formats can preseve extended attributes already - e.g. tar already does.

probonopd commented 3 years ago

Uh. I am not a huge fan of patching half the system. Maybe better use dot files in each directory. E.g, .extattr? When Filer sees such a file, it could read it and apply the extended attributes in the filesystem. Something along those lines (not fully thought through yet).

ghost commented 3 years ago

Yeah, OK, I think you're right.

We'll need a spec - doesn't have to be fully fleshed out, but if we can get a standard for naming to start with that would be good for modifying the spatial Filer PR (as a prototype).

All files/folders will need a dot file I think, rather than a just one for the directory or you have all sorts of management of what then becomes like a flat database.

e.g.:

/home/chris/projects
/home/chris/.extattr_projects
/home/chris/projects/document.odt
/home/chris/projects/.extattr_document.odt

The dot files can contain key value pairs for the attributes, although I'm not sure about the best format - possibly just key=value one per line?

It doesn't need to apply the extended attributes at all really, as the dot files will need to be kept up to date - may as well just use the dot files?

ghost commented 3 years ago

OK, I see the problem with that - the directory needs to contain its own dotfile or you'll lose it on archiving.

So:

/home/chris/projects
/home/chris/projects/.extattr
/home/chris/projects/document.odt
/home/chris/projects/.extattr_document.odt

probonopd commented 3 years ago

Instead of having

/home/chris/projects/.extattr_document.odt

we could have that information in

/home/chris/projects/.extattr

as well. Only directories would need those files, the files could also describe the documents inside. When Filer sees the directory, it would apply its extattrs to the files inside.

ghost commented 3 years ago

It won't scale for very large numbers of files in a directory I think

probonopd commented 3 years ago

People are gonna hate this if we have such files cluttered around in big numbers.

Actually, let's have a look at how .DS_Store files work. They are created by the Finder in some circumstances and seem to contain the metadata we are discussing here. The format is proprietary, but maybe we can deduce some of the "theory of operation".

probonopd commented 3 years ago

ghost commented 3 years ago

Looks a bit complicated!

We might want to consider something like sqlite?

probonopd commented 3 years ago

Our motto: Aim for making everything 10x as simple! But it cannot hurt to understand the "prior art" concepts first.

We might want to consider something like sqlite?

Sure. But we need to think about what to put where, so that it can travel along, e.g., in zip files and such. And we need to think about performance. The Wikipedia article has some good hints. (To interpret them liberally: By default do not store such files on network shares, and only save those files after a window was open for a certin number of seconds and the user changed its appearance - something like that)

ghost commented 3 years ago

Sure. But we need to think about what to put where, so that it can travel along, e.g., in zip files and such. And we need to think about performance. The Wikipedia article has some good hints.

I'm suggesting that the single .extattr file in the directory is an sqlite database, so it would persist with the zipped directory. But yes, not sure about performance.

probonopd commented 3 years ago

Yes, that would be worth a try.

probonopd commented 3 years ago

Check this out!

https://ds-store.readthedocs.io/en/latest/

Manipulate Finder .DS_Store files from Python. (...) This module allows programmatic access to and construction of .DS_Store files directly from Python, with no Mac OS X specific code involved.

Wouldn't it rock if we could use the same format even, given that someone seems to already have written the implementation?

Reading through the comments in the code of https://github.com/al45tair/ds_store/blob/master/ds_store/store.py alone is versy interesting, it gives some hints such as:

Works by manipulating the file on the disk---so this code will work with .DS_Store files for very large directories.

I'd say let's play around with https://github.com/al45tair/ds_store a bit and in case we decide against using it we know at least why ;-)

antranigv commented 3 years ago

Please let’s not use DS_Store, as a former macOS user it was a PITA for me, specially when doing development.

This means it will be very useful for the daily users, but very bad experience for the power users and developers.

On Sat, Feb 20, 2021, at 3:03 PM, probonopd wrote:

Check this out!

https://ds-store.readthedocs.io/en/latest/

Manipulate Finder .DS_Store files from Python.

Wouldn't it rock if we could use the same format even, given that someone seems to already have written the implementation?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/helloSystem/hello/issues/79#issuecomment-782607064, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKXL3P5TT6XEZ5X37FO4LTS76JIVANCNFSM4VQPWWMQ.

probonopd commented 3 years ago

The format seems to be documented here: https://metacpan.org/pod/distribution/Mac-Finder-DSStore/DSStoreFormat.pod

What were your gribes with .DS_Store fies? I am asking because I fear that some of them will be there with any such format.

At the very least, we should understand .DS_Store before we implement something new.

probonopd commented 3 years ago

At the bare minimum, https://metacpan.org/pod/distribution/Mac-Finder-DSStore/DSStoreFormat.pod tells us conceptually which type of information we should consider to store. If we then find a much simpler way, all the better!

antranigv commented 3 years ago

My issue was not with DS_Store specifically, but with the concept that there’s a dot file in each directory which is tightly integrated with Finder.

If I did mv/cp manually from the CLI, then DS Store would not catch that. And I had to ignore it every time I did development either with git or svn.

Again, I think for the end user it’s the best thing out there, but one level up (power-user) and it starts to be a pain.

On Sat, Feb 20, 2021, at 3:08 PM, probonopd wrote:

The format seems to be documented here: https://metacpan.org/pod/distribution/Mac-Finder-DSStore/DSStoreFormat.pod

What were your gribes with .DS_Store fies? I am asking because I fear that some of them will be there with any such format.

At the very least, we should understand .DS_Store before we implement something new.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/helloSystem/hello/issues/79#issuecomment-782607710, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKXL3NA5FRZ5F6RK7SDC3DS76J4DANCNFSM4VQPWWMQ.

ghost commented 3 years ago

So would extended attributes be better, even if you might lose spatial information on transfer to some file systems or network?

ghost commented 3 years ago

Note, that unlike with Haiku (who use their own BeFS attributes), we don't have to patch archivers and network file systems for support.

With extended attributes, it looks like tar, zip, NFS all have support - we are in a better situation than Haiku with this.

You can test this now - with https://github.com/helloSystem/Filer/pull/35 we are using extended attributes already, so we can have a play with packing/unpacking etc to see where the issues are

antranigv commented 3 years ago

So would extended attributes be better, even if you might lose spatial information on transfer to some file systems or network?

I think that's fine, since that's how it work in other systems as well.

I have not done a full research here, but extended attributes seem very nice compared to having a dot file in every directory.

On the other hand, as @probonopd noted, we need to handle "image" installers as well, but that seems to be doable if we zip things, right? or we would loose the attributes if we distribute things over the net? hmm this needs trial and error to understand better.

probonopd commented 3 years ago

My issue was not with DS_Store specifically, but with the concept that there’s a dot file in each directory

That will probably be unavoidable if we want metadata to travel in a safe way with a directory...

I know that the feature will not be for everyone so maybe we should make it possible to deactivate the file generation. In this case, if the extended attributes are lost, so is the metadata.

ghost commented 3 years ago

OK, so I think we should definitely have a dot file mechanism in addition to the extended attributes.

I tried the FreeBSD zip/unzip and it either doesn't have the patches or support for preserving the extended attributes.

Have a preference to turn on/off writing to the dot file (but always attempt to read/write to extended attributes). I will look at how the performance of sqlite is for this (next weekend maybe).

I think .DS_store files we could look at adding maybe just read support later on by reusing an existing parser somehow? Essentially, as legacy support.

Does that sound like a good plan @probonopd ?

probonopd commented 3 years ago

Indeed. Since people do not universally like the presence of .DS_store files I was thinking, why add yet another such file if we could use the ones that may already be there...

probonopd commented 3 years ago

We also need to do a systematic check whether all functionality can be reached in spatial mode, e.g.,

Unmounting devices
Going "up" in the hierarchy (is currently possible via the menu but we need Cmd-ArrowUp to work, and it should open the correct spatial window rather than changing the contents of the current one)

bitigchi commented 3 years ago

We also need to do a systematic check whether all functionality can be reached in spatial mode, e.g., unmounting devices.

All mounted drives, devices, networks should be available through a 'Go' menu.

Also, please don't do .DS_Store files. They are a PITA while dealing with git and other operating systems. I think Haiku uses extended attributes for the window properties, but if it's not possible, there are talks for a special workaround for the read-only folders. Maybe something similar can be utilised.

ghost commented 3 years ago

We also need to do a systematic check whether all functionality can be reached in spatial mode, e.g.,

Unmounting devices

Going "up" in the hierarchy (is currently possible via the menu but we need Cmd-ArrowUp to work, and it should open the correct spatial window rather than changing the contents of the current one)

Yep, Haiku has an 'open parent' menu item which is also Cmd-ArrowUp.

It would be nice if all devices could show up on the desktop and be ejected from there (drag to trash?)

ghost commented 3 years ago

Also, please don't do .DS_Store files. They are a PITA while dealing with git and other operating systems. I think Haiku uses extended attributes for the window properties, but if it's not possible, there are talks for a special workaround for the read-only folders. Maybe something similar can be utilised.

We are using extended attributes already (just window position and size at the moment). The dot file suggestion was because there are locations the attributes won't persist - e.g. some archives, network shares, cloud sync etc. And the plan is to use both but have an option to just write the attributes in the preferences, so you could turn off the annoying dot files.

I'm not 💯 convinced it's the right solution yet anyway. I was thinking that we could at least make whatever GUI archiving tool we supply preserve extattrs, and add requests that NextCloud handles them etc.

The read only areas we already have an idea about how to solve, similar to the one suggested in that bug report.

ghost commented 3 years ago

@probonopd BTW, I raised another PR at the weekend for Linux support of the extended attributes:

https://github.com/helloSystem/Filer/pull/37

probonopd commented 3 years ago

It would be nice if all devices could show up on the desktop and be ejected from there (drag to trash?)

Definitely. Check https://github.com/helloSystem/Filer/issues/9 for some details.

Ejecting should also be possible via the right-click context menu.

ctrlcctrlv commented 3 years ago

I'm very excited for this. Back to basics, it'll be usable on Linux, right?

ghost commented 3 years ago

I'm very excited for this. Back to basics, it'll be usable on Linux, right?

Yeah the spatial mode works on Linux too, which is nice because I use xubuntu on my dev machine for my day job (just replace Thunar in the session). Obviously it's better in helloSystem! 😀

I'm not sure what @probonopd thinks about how integrated it should be with FreeBSD in future and as to whether certain features won't work on Linux.

ghost commented 3 years ago

I have created a new PR to fix the operation of all the items in the 'Go' and 'Bookmarks' menu now in spatial mode, so, e.g. alt-up will open parent in its own window (with remembered position/size):

https://github.com/helloSystem/Filer/pull/40

probonopd commented 3 years ago

Thank you very much @moochris, this is really great work. :+1:

I'm not sure what @probonopd thinks about how integrated it should be with FreeBSD in future and as to whether certain features won't work on Linux.

It would be good if we could keep it in a state that allows it to compile on Linux, but we don't necessarily need to implement all functionality for Linux immediately; I am fine with printing out "TODO: Not yet implemented for Linux" in places where we have a FreeBSD-specific implementation of something. This way, should someone come along who really cares deeply about running Filer on Linux it's possible to implement the missing platform-specific pieces.

probonopd commented 3 years ago

e.g. alt-up will open parent in its own window

Do you have a working shortcut there?

ghost commented 3 years ago

e.g. alt-up will open parent in its own window

Do you have a working shortcut there?

Yes - that's weird. Perhaps the translations have some bearing on the keyboard shortcuts? It's been a while since I looked at the Qt translation stuff.

filerscreenshot

probonopd commented 3 years ago

Yes, definitely; I need to copy the translations in place :)

Also, since we are swapping Ctrl and Alt system wide, I think we should set Ctrl-Left/Right/Up as shortcuts to get the intended end result...

helloSystem / hello

Spatial file manager #79