Live community hangout #5 - Sept 21st, 10AM PST

[okdistribute]

We will have our fifth community call next Monday using Hangouts on Air! The entire community is encouraged to join and listen in, ask questions, and see where we're going next with dat.

Youtube Link: http://youtu.be/rj71iaugcXA Gitter link (chat/ask questions): https://gitter.im/datproject/discussions

Theme: On api simplification and containerization Where: Google Hangouts on Air Date: Monday, September 21st. Time: 1pm EST (17:00 GMT)

Leave your questions in the thread below, in our gitter web chatroom or #dat on freenode

Agenda:

• Progress on containerization - discuss integrating http://hyperos.io/ with the dat CLI
• 1.0 CLI API discussion -- making files first-class citizens
• mdns names api
• Progress on dathub
• Progress on schemas

We'll be using Hangouts On Air.

Note that only Dat team members and others on the agenda will be invited to join the Hangout so they can broadcast voice and video. Hangouts only supports about 10 people max, but the youtube stream can support an unlimited amount of viewers.

[max-mapper]

some questions related to https://github.com/maxogden/dat/pull/403 and CLI API issue above

• what does the step-by-step workflow look like when you clone, checkout a file, modify it, add your changes, push?
• does dat add expect to 'sync' the working directory with the files in dat? e.g. how git works
• e.g. if you dat add *, modify some, dat add * again, what output do you see?
• should files get checked out by default into the working directory when you clone/checkout? currently requires duplicating file into working directory (because no copy-on-write/fuse). maybe move this to a separate command? having hyperos impacts the importance of this one...
• should we revisit the concept of subfolder-per-dataset and have multiple files datasets?
• what relevant constraints are imposed and/or new features are unlocked when we integrate hyperfs with dat?
• what needs tests/docs? (e.g. hyperfuse/hyperfused). lets create a big markdown table

other random notes for call (disregard unless you're me, doesnt make much sense)

[ ]
[third floor: https://github.com/mafintosh/hyperfs] (filesystems on top of graphs)
[second floor: https://github.com/mafintosh/hyperlog] (graphs on top of logs)
[first floor: https://github.com/mafintosh/level-logs] (append only logs)
[ground floor - node.js + leveldb]

in hypercore: edit file -> mac os -> hyperfs -> new edit to existing file
  e.g.: 'echo "foo" >> bar' -> only be a diff on top of existing bar
in mac os: edit file in dat -> replaces existing file
  e.g.: 'dat put foo.jpg' -> completely replaces existing foo.jpg

dat clone
ls

image.zip                           1GB

{
  "image.zip": [
    "2039j23j402",
    "j2034j2093jj",
    "23j420394j20"
  ]
}

dat add image.zip
-> 1mb "2394u02893j" >> image.zip chunk list

dat sync
^ 1mb

=======

- download all metadata in graph
- download set of files at version, only latest copy of each file, into users working directory as correct names ready to use (no previous file version data)
- download all old versions into .dat folder
- just update graph to see if new metadata is available, dont touch working directory

dat clone foobar.com
dat checkout
dat sync
dat fetch

notes from call

dat init
dat commit "hi"
dat push foo.com

- client pushes everything
- server gets everything
- client only stores metadata afterwards

dat init
dat snapshot "hi" # commits + backs up files into .dat
dat push foo.com

- client pushes everything
- server gets everything
- client has everything

[okdistribute]

I'd love to start with 'what relevant constraints are imposed/new features unlocked when we integrate hyperfs with dat.' If there's a more elegant way to think about this -- especially if we can get deduplication to work within bytes of blobs -- I'm all ears.

[max-mapper]

@cshum oh sorry I just realized we scheduled it for 1AM your time again. maybe the next one we can do 8AM your time/5PM our time

[sallespro]

hi, i did catch a bit of the conversation today, but missed the hyperfs / containers big picture. in order to enable the code joining the data keeping file archives as primitives, what does it look like to play with dat containers ?

i had to take the way of using https://github.com/zchee/docker-machine-hypercore but did't really understand if we're working this way after all.

[okdistribute]

@sallespro we mostly punted on coming up with a solid commandline interface for dat containers right now. Towards the end we start getting into the weeds, if you want to take a ganter. :) What's your use case with docker-machine-hypercore?