search

holepunchto / hyperdrive

Hyperdrive is a secure, real time distributed file system
Apache License 2.0
1.86k stars 135 forks source link

Undocument `drive.clear` #315

Closed mixmix closed 1 year ago

mixmix commented 3 years ago

Hi I've been working on how to remove files from my hyperdrive. Have looked across the some issues and this PR and seems like drive.clear is what I'm looking for

couple of questions:

  1. any reason this is undocumented? (other than the usual - time!)

  2. can I use hyperdrive.clear with storage: random-access-file?

I see random-access-file has a file.del which is likely called by hypercore.clear, but raf also says of file.del

"Will truncate the file if offset + length is larger than the current file length. Is otherwise a noop."

To me this suggest that unless I'm deleting the most recent files, nothing will happen? I'm running some tests and not sure I'm seeing my storage folder reducing in size after a clear (using linux du -d 1 storagePath)

(apologies this question spans repos, I'm still learning the stack)

mixmix commented 3 years ago

cc @okdistribute - I'm testing code magma collective wrote. I think they might have actually found about this API from you!

mixmix commented 3 years ago

excitingly I have file which has different sizes based on how you ask now:

➜ ls -lah a3/8f/a38f5d6a076fbfb7d1830886822aad4de0b7feb0de8b96b255d277a5a4f49995/data 
-rw-rw-r-- 1 mix mix 2.2G Jun 21 15:41 a3/8f/a38f5d6a076fbfb7d1830886822aad4de0b7feb0de8b96b255d277a5a4f49995/data

2.2GB

➜ du -h a3/8f/a38f5d6a076fbfb7d1830886822aad4de0b7feb0de8b96b255d277a5a4f49995/data
901M    a3/8f/a38f5d6a076fbfb7d1830886822aad4de0b7feb0de8b96b255d277a5a4f49995/data

901MB

Reading around I'm learning about "apparent disk size", as in file size !== disk usage. If I've read this right we have a very large sparse file (has lots of empty zeros) so while the "apparent size" is 2.2GB it's only actually taking up 901MB of space currently?

Having a hard time knowing what to read to report disk usage to users (and to assess how pruning files is going)

RangerMauve commented 3 years ago

I think it's undocumented because nobody got around to documenting it. 😅

I've been using it in production and it seems to work decently with random-access-file.

Sparse mode is more stable on Windows as of this release: https://github.com/random-access-storage/random-access-file/pull/24

mixmix commented 1 year ago

Documented here : https://github.com/holepunchto/hyperdrive#const-cleared--await-driveclearpath-options