Storage utility (or disk utility …)

grahamperrin commented 3 years ago

I'd like something like:

Apple's Disk Utility, as it was before the post-Yosemite redesign; or
GParted

– I can't expect GParted itself on a FreeBSD-based system.

probonopd commented 3 years ago

Indeed! This is something that needs to be added.

Do you know a good one written in Qt or PyQt5 or do you think we'd need to write it from scratch?

Ideally it should also expose the most commonly used ZFS features in a human-friendly way.

probonopd commented 3 years ago

Prior Art:

https://docs.kde.org/trunk5/en/extragear-sysadmin/partitionmanager/usermanual.html Pro: Powerful, Cons: KDE dependencies, too many technical details, possibly does not use gpart under the hood
deepin-diskmanager Pro: Appears to be simple, Cons: DDE dependencies, no Qt native look (e.g., menu in weird hamburger menu), probably does not use gpart under the hood

At lest they can provide some inspiration for the code to be written?

grahamperrin commented 3 years ago

I vaguely recall something in Lumina, so long ago that it might have been before TrueOS. Certainly so long ago that I can't recall the UX. I do recall that it was somehow buggy (this is not a criticism; it was early stuff). Postscript (note to self): was it maybe something like a system tray notification area pop-up that was primarily for connecting and ejecting volumes, secondarily for management of partitions and/or file systems?

I installed x11/lumina today … if I don't find the utility in this environment, I'll dig through my archives and maybe find a point of reference.

Commentary crossing paths: above, that's interesting, thanks.

probonopd commented 3 years ago

On the simplistic end:

https://github.com/Liemaeu/FreeBSD-USB-Quick-Formatter

maybe something like a system tray notification area pop-up that was primarily for connecting and ejecting volumes, secondarily for management of partitions and/or file systems?

No. It must "just work" like on the Mac. No notifications and tray icons needed.

probonopd commented 3 years ago

What I think made Mac OS X Disk Utility (when it still was good) stand out was that it arranged information and capabilities that are typically spread out across several tools.

Where would the information come from?

1: /sbin/geom disk status -s, /sbin/geom disk list <geom>
2: Recently used disk images
3: /sbin/geom disk list <geom>
4: /sbin/gpart show -lp <geom>
5: dd and friends? Are there tools that deal with disk images specifically? qemu-img,... (qcow2 seems to be a nice format because it can hold any filesystem/partition scheme and still use zlib compression...)
6: growisofs

grahamperrin commented 3 years ago

Note to self: find screenshots of what was in older releases of Mac OS X Server. Inspirational.

probonopd commented 3 years ago

Also fun: Run Mac OS X Public Beta, see how they cobbled everything together from here and there (clearly visible before everything was unified and polished).

grahamperrin commented 3 years ago

https://github.com/helloSystem/hello/issues/61#issuecomment-751340339

… No notifications and tray icons needed. …

Agreed. (That part of my note to self was purely to help recall what it might have looked like, if I have to dig deep through archive material.)

grahamperrin commented 3 years ago

From https://forums.macrumors.com/threads/os-x-el-capitans-disk-utility-is-a-mess.1894047/page-7?post=22209602#post-22209602:

In particular:

no attempt to crowbar belongings into the sidebar.

probonopd commented 3 years ago

It's not that I think Apple is the gold standard in all cases. Let's see whether we can make things even simpler.

And of course make full use of what FreeBSD gives us. For example, instead of "RAID" I was thinking of an "OpenZFS" tab maybe?

What would it need to be able to do?

probonopd commented 3 years ago

This is what I have come up with so far. Just sketching it out in PyQt5 using Qt Creator and some Python to call /sbin/geom and friends.

You can download and run it from this repository, but besides displaying information about disks no functionality has been implemented yet. However, anyone with Python skills could probably fill in the rest from here, as the basics have been sketched out.

Volunteers?

probonopd commented 3 years ago

Constantly thinking about how to simplify:

Do we really need (2) when there is (1)? Isn't this redundant? Can we intelligently combine the two?
So far, we have disks -> partitions. With zfs, wouldn't we have another layer, namely disks -> partitions -> zfs datasets? Not really, because a dataset resides on a pool, and a pool can be made up of any number of disks and files. How to represent this in the most natural way? Would pools need to appear in (1), so that their datasets could be edited in (2)?

probonopd commented 3 years ago

(1) shows the status on disk whereas (2) shows a potential future state after the user has clicked "Apply".

But then: Do we really need that?

I am not a fan of "Apply" buttons, and they are inconsistent with the rest of the system, where changes get applied and actions get executed immediately.

probonopd commented 3 years ago

Display zpools alongside physical devices?

And their datasets similar to how partitions are displayed for physical devices?

probonopd commented 3 years ago

https://twitter.com/yukkuriyankey/status/1343314065293078530

Can you derive any inspiration from freenas' UI? It seemed pretty sensible with regards to such things IIRC.

Good idea.

grahamperrin commented 3 years ago

+1

I haven't used TrueNAS core yet but I do recall the web interface to FreeNAS being reasonably user-friendly for management of volumes, file systems etc..

grahamperrin commented 3 years ago

Ideally design the utility in a way that will make it easy to avoid clobber with OpenZFS:

https://github.com/helloSystem/hello/issues/32#issue-730983594 (opening post)
https://github.com/helloSystem/hello/issues/32#issuecomment-752016705 (2020-12-19)

Visualising the utility as a complement to Filer (https://github.com/helloSystem/Filer)

Historically: ZEVO Community Edition and/or its predecessor Z-410 Storage allowed snapshots to be browsed in Apple Finder. I don't have screenshots handy but the integration was very neat.

One bug was memorable, I can't recall the exact details but symptoms included unwanted automatic mounting of snapshots (something like that) and IIRC it was worked around by not using Finder to browse following the mount. So, for example, it was OK to browse using Terminal.

With more recent OpenZFS on OS X:

… Currently mounted snapshots are only visible from Terminal, not from Finder. …

– whether that's by design, I don't know, but I'm reminded of the Terminal-oriented workaround with ZEVO.

Wonder whether things such as importing OpenZFS pools without mounting, and temporary mount points, should be within the UX for a storage utility; or the UX for Filer.

Gut feeling (early/experimental): storage utility.

(Be wary of crow-barring anything too complex into Filer.)

probonopd commented 3 years ago

snapshots to be browsed in Apple Finder

https://github.com/furybsd/furybsd-livecd/issues/215

probonopd commented 3 years ago

Indeed "Storage" may be the better word than "Disk Utility", once it deals with more than disks.

grahamperrin commented 3 years ago

https://github.com/helloSystem/hello/issues/61#issuecomment-751338205

… Ideally it should also expose the most commonly used ZFS features in a human-friendly way.

+1

Re: https://old.reddit.com/r/freebsd/comments/klvrwd/bsd_users_on_twitter_this_week/ghdow08/

At its simplest (and safe, with encryption), OpenZFS can be like this:

$ lsblk da0
DEVICE         MAJ:MIN SIZE TYPE                              LABEL MOUNT
da0              2:59  466G GPT                                   - -
<FREE>         -:-   1.0M -                                     - -
da0p1          2:60  466G freebsd-zfs                gpt/efiboot0 <ZFS>
$

In a little more detail

I call what's below my t500 volume because it's on a 500 GB mobile hard disk drive, a StoreJet Transcend. t for Transcend.

I could have simply encrypted everything but instead, I left the primary file system unencrypted. I can't predict what use I'll make of it in the future.

I knew from the outset that I'd want my VirtualBox data encrypted, so I created an encypted file system that mounts at /Volumes/t500/VirtualBox

A ranger view of the VirtualBox folder:

A command line view of relevant properties:

$ zfs get mountpoint,compression,compressratio,encryption,encryptionroot Transcend Transcend/VirtualBox
NAME                  PROPERTY        VALUE                     SOURCE
Transcend             mountpoint      /Volumes/t500             local
Transcend             compression     zstd                      local
Transcend             compressratio   1.59x                     -
Transcend             encryption      off                       default
Transcend             encryptionroot  -                         -
Transcend/VirtualBox  mountpoint      /Volumes/t500/VirtualBox  inherited from Transcend
Transcend/VirtualBox  compression     zstd                      inherited from Transcend
Transcend/VirtualBox  compressratio   1.59x                     -
Transcend/VirtualBox  encryption      aes-256-gcm               -
Transcend/VirtualBox  encryptionroot  Transcend/VirtualBox      -
$

For a 'simple' user of helloSystem

You might wizard up two choices when (for example) preparing a USB flash drive for use with the computer:

safe but not compatible with macOS or Windows – require creation of a password, and make clear that there's no forgotten password routine (behind the scenes: a single encrypted file system)
compatible but not safe (behind the scenes: 0700 (Microsoft basic data)).

grahamperrin commented 3 years ago

FYI https://redd.it/l0d0mv ▶ https://github.com/manoeldesouza/zyggy

@probonopd I know, you don't want GTK, but looking at the feature sets of comparable utilities can help to shape thoughts around what to include in a helloSystem storage utility.

Re: the feature set of zyggy, some overlap with https://github.com/helloSystem/Utilities/issues/155

probonopd commented 3 years ago

Instead of writing a full blown Disk Utility, I will implement a simple "Format Disk" tool first:

helloSystem / hello