Brig unable to push changes

[jayalshah]

Shows that difference exist between the files but not able to push.

It does not show any kind of error after executing the push command but left with no result.

[evgmik]

Hi, I can confirm broken sync with brig v0.4.1 and ipfs v 0.5.1.

brig diff rmt shows some diff indicated with ⇄, for example

•
└──testdir
   └── t ⇄ t

0 directories, 1 file

the run of brig sync reports

Syncing with `rmt`...
Nothing changed.

A consequent run of brig diff rmt shows the same diff as before.

Non conflicting changes seems to propagate just fine.

[datatypevoid]

I am experiencing this with v0.4.1 on ipfs 0.7.0. Anyone have any insight?

brig version output:

           Client Version: v0.4.1+68f8766
               Client Rev: 68f8766fd9fe8929e8b3fc6cefca01454f380a5b
           Server Version: v0.4.1+68f8766
               Server Rev: 68f8766fd9fe8929e8b3fc6cefca01454f380a5b
   Backend (ipfs) Version: 0.7.0
       Backend (ipfs) Rev: 
               Build time: 2019-03-31T00:10:46+01:00

[evgmik]

I made quite a lot of patches in my clone of brig https://github.com/evgmik/brig which, as I think, fixed this issue and several others. As of now I am not actively working on brig: the IPFS has pitiful throughput to make it suitable for large dataset updates. So do not expect much help from me there. But at least I worked hard to make syncing consistent.

I wish @sahib would come back to this project, I liked the idea quite a lot and implementation was quite solid.

[sahib]

Hello @evgmik,

thanks for your work. I will look at your commits in the next days and try them out. I have a few free days over christmas now and might work a bit on brig - mostly to see if IPFS performance changed by now.

I wish @sahib would come back to this project, I liked the idea quite a lot and implementation was quite solid.

I wish that too. :smirk: But time constraints - software engineering as a day job doesn't particularly motivate you to also spend your evening in front of a computer. But it's on my new year's resolution list to work a bit more on my FOSS stuff...

[evgmik]

This would be fantastic @sahib!

If you are back, I would love to push the project further. Just a note, I did some changes in the Repinner logic. The goal was never to unpin a file which is pinned. I never made it through, but repinner should somehow check the global (across network) pin status, otherwise all replicas might unpin a file and it will be gone everywhere. I was trying to use brig for important files to sync them, but chance of loosing it all was a show stopper.

As for IPFS throughput, there is a builtin limit due to protocol which was there in version 0.6 and probably will be there forever. https://github.com/ipfs/go-ipfs/issues/5083 Since all files are split in chunks of let's say 256kiB the transmission is goes in chunks as well, so you have to renegotiated transmission every 256kiB, with round trip delays. So throughput is limited by round trip time even on very fast networks: 256kiB/(round trip time). For my case 256kiB/30ms about 1MB at max, there is also issue of getting same chunk from other nodes at the same time. If you have more distribution peers it would go up, but my usual case is office and laptop connection, so only 1 peer is possible.

[sahib]

If you are back, I would love to push the project further.

Great to hear! Developing brig solo was pretty daunting. Maybe we can slowly, but steadily make it to some useful state.

Just a note, I did some changes in the Repinner logic. The goal was never to unpin a file which is pinned. I never made it through, but repinner should somehow check the global (across network) pin status, otherwise all replicas might unpin a file and it will be gone everywhere. I was trying to use brig for important files to sync them, but chance of loosing it all was a show stopper.

Yes, this was on the roadmap (see last point) already. I would be very glad to see that feature too.

As for IPFS throughput, there is a builtin limit due to protocol which was there in version 0.6 and probably will be there forever. ipfs/go-ipfs#5083 Since all files are split in chunks of let's say 256kiB the transmission is goes in chunks as well, so you have to renegotiated transmission every 256kiB, with round trip delays. So throughput is limited by round trip time even on very fast networks: 256kiB/(round trip time). For my case 256kiB/30ms about 1MB at max, there is also issue of getting same chunk from other nodes at the same time. If you have more distribution peers it would go up, but my usual case is office and laptop connection, so only 1 peer is possible.

Okay, interesting. I noticed the small download speeds, but that sheds some more light on it. That of course defeats a bit the purpose of brig as "git for big files".

---

Maybe we should move that conversation to the matrix chat room? Or do you prefer another way of communication?

I also looked over the changes you did and they look good so far :+1: - I noted down a few things that could be improved, but those are mostly details. Tomorrow I will merge your changes to my develop and try them out. Next step is for me to improve a few Go/build related things, do general cleanup and remind myself how brig actually works :smile:

[evgmik]

I would prefer an email, I am not to fluent with new trends: matrix, discords, etc. I will send you an email to the address which you list at github. If we need a real time communication, I can always learn them, but for now email saves me from one more thing to install.

When you do the pull, remember I broke one of the rules of patches for brig: write a test case. I think I did make a couple, but it was a bit beyond my understanding of go. Essentially, I learnt go when I was working on brig.

[evgmik]

I did some benchmarks IPFS 0.7 is still slower than rsync but at now it much closer to saturating my link throughput.

• rsync does 3.5MB/s
• ipfs 0.7 does 2.5 MB/s