IPFS Implementation - C++, Any Interest in a C++ Protocol Implementation?

[kevina]

Is there any interest in a C++ protocol implementation (not just an API client library) either now, or at some point in the future?

I image that if you want to eventually get IPFS into the O.S. you will likely need a pure C implementation (at least for Linux). A C++ implementation might be a good starting point.

If you want to get a protocol implementation into the browsers a C++ implementation might help. It might also help with mobile.

The C++ implementation might be faster than the reference Go implementation.

Since I am just getting up to speed in IPFS I am mainly interested the core developers opinion on this matter.

[whyrusleeping]

I would be interested in having this. The starting point for any new ipfs implementation is writing an implementation of libp2p in that language. Even if we end up deciding we don't really want a full implementation of ipfs in C++, having libp2p will be awesome for the project.

[klartext]

If an implementation in C seems to be the way to go, why writing in C++ first instead of C?

... if other implementation languages would be of interest, I would propose OCaml :-)

Looking at the go-libp2p-lib, the ipfs-network-spec-link yields a 404-error. Hmhhh, I would guess thats the most important document though....

[whyrusleeping]

re the broken link, sorry about that. It looks like someone renamed things over in the specs repo.

Here is the correct link: https://github.com/ipfs/specs/tree/master/libp2p (I'll also update over in the libp2p readme)

[klartext]

...is there also a one-document (pdf?)-version?

[whyrusleeping]

nope, parts of it are still somewhat in progress so we havent yet compiled it all together.

[kevina]

@klartext I have no idea if an C implementation is the way to go. At his point I am just trying to get the feel for what the core developers are thinking in terms of long term plans.

A C++ implementation will be easier to write (at least for me). As I see it, is is unlikely that the entire protocol belongs in the (Linux) kernel; the parts that need to be in the kernel can likely easily be rewritten in C once you have C++ code to base it from.

[lidel]

I don't think you should think about C as "something that is useful only around kernel development". It is useful in user land too, for multiple reasons. Short opinion below (skip if not interested) :-)

From my experience, when writing systems code such as IPFS, you may get a bigger bang for the buck with C:

1. It is more portable. C++ compilers exist only for the most popular platforms, while C compilers exist for just everything, even the most exotic hardware.
2. Note that most other languages out there have natural bindings to C (Python, Lua, Go and others support it out-of-the-box). I've met with an opinion that C++ is a huge pain to interface with, especially at the binary level.
3. C++ complexity makes collaboration difficult (a bigger problem than it seems, especially in open source):
   • When two programmers tell you "I can code C++", they might both know 20% of the language, however a different 20%, so they would not be compatible with each other.
   • C APIs are more alike and easy to grasp. C++ libraries are often like they come from separate planets (compare Boost and Qt).

In short C++ would be "just another implementation" (which is still nice to have), but C implementation could improve IPFS portability by a lot.

[whyrusleeping]

I'm quite familiar with C and would be amenable to working on such a project. Anything done in C++ is going to be either 'C with classes' or something I don't understand very well.

Also worth noting is compile times, theyre one the reasons I love go. C's compile times are pretty good if you write your makefiles correctly, but C++ compile times leave plenty of room for office chair swordfighting

[montagsoup]

I would love to see a C implementation. It would really open up development in certain areas. One use case I thought of a while back was a game with resources hosted on IPFS (this gets rid problem of effectively hosting and downloading some 25 GiB MMO). A lot of games are written in C/C++ so Go wouldn't be the best option for them. As for whether C or C++, I think plain C would be better. Allows for a wider audience (both C and C++ instead of just C++).

[kevina]

@lidel it is very easy to create a C API for a C++ program and bind to that. (In fact in one of my projects (Aspell) I only export a C API to avoid many of the problems with a C++ API.) Linking pure C code with a C++ library on most system does not create a problem as long as shared libraries are used. For this reason I also believe a C++ implementation, while not as useful as a C one, will be more useful than "just another implementation" as it will create code than can be linked to from any application on most platforms.

@whyrusleeping in my experience C++ compiler times are not that big of a concern, especially on small to medium projects.

I am certainly capable of writing something in pure C. However, I can probably create a C++ implementation in a lot less time then I could A C one. I will also be far more confident the C++ program is free from memory errors than I would if I had to write a pure C one.

[Kubuxu]

@kevina it is quite easy to use C API in C++ program but it doesn't make a lot of sense then. C API won't use smart pointers, collections and so on.

Also most people are not familiar with new C++ some of them even dislike how bloaty it has become.

IPFS isn't small project (170k LOC with dependencies), and it will grow as time progresses, so compile times are concern.

There are ways to solve memory allocation problems in pure C, one of those ways is to use exclusively custom memory allocation method. This is what cjdns does and it really works. Development time in C might be longer at first but when codebase establishes it is much better than C++, error hinting works much better in C (ever made error in template in C++ ?) build times are shorter which is really important for debugging and basic OO coding is possible, again approach taken by cjdns, down- and safe up-casting with runtime check for find possible errors.

@montagsoup go can be linked with C/C++, it requires writing bindings and using gogcc but is viable.

[montagsoup]

@Kubuxu Of course go can be linked to C. Tons of stuff can, and I think that's another good point in the favor of a C implementation. Not only would it open up C development, it would open up development for any language that can link to C functions.

I'd be willing to help with a C implementation as much as I can if that's a direction people start deciding they want to go.

[xsnk]

How about Rust?

On Wed, Mar 23, 2016 at 2:41 AM, montagsoup notifications@github.com wrote:

@Kubuxu https://github.com/Kubuxu Of course go can be linked to C. Tons of stuff can, and I think that's another good point in the favor of a C implementation. Not only would it open up C development, it would open up development for any language that can link to C functions.

I'd be willing to help with a C implementation as much as I can if that's a direction people start deciding they want to go.

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/ipfs/ipfs/issues/164#issuecomment-200030246

[notabeatle]

I was surprised to find that the core of the reference IPFS implementation was not written in C. Most projects of this sort seem to go that way so they get Python/Ruby/Rust/PHP/Go/et c libraries—not to mention plugins to or modules for other major projects—practically for free and so, as infrastructure on which other projects may depend, they don't set the performance ceiling for such dependent projects any lower than absolutely necessary. At first glance I figured Go was just wrapping a C lib to act as the command line interface to it. That this is not the case remains my greatest concern for the long-term health and prosperity of this project, FWIW.

[whyrusleeping]

I used to work on distributed systems in C, now I work on distributed systems in Go and I'm much happier.

[dcbishop]

I'm not an IPFS dev so I might be missing something here but...

@a-shark: IPFS is run as a daemon, so I'm not sure how relevant that is since you talk to it over networking protocols.

Having C-ABI compatibility is quite useful for libraries. There are heaps of really awesome libs that will never get used outside of their programming languages ecosystem. At least not without some non-standard bridge between any two languages pairs.

C is used for writing the OSes, languages making syscalls need compatibility with the C-ABI in that platform so basically all languages support the C-ABI at some level.

But other languages support generating C-ABI compatible libraries (Go, Rust, D, C++ all seem to).

As for if a program needs to be written in 'C' can be an problem on platforms that either don't support the compiler. Or ones that are resource constrained to the point that some of the language features such as the garbage collector are an issue. That could pop up if people are trying to put IPFS into routers and similar embedded devices. I'm not sure how well IPFS would fit into those kind of systems, at the very least they would need somewhat storage to store the files they are hosting.

Since Go 1.5 you can build C callable libraries. -buildmode=c-archive or -buildmode=c-shared. Blog post.

It's also worth considering that some languages can do source-to-source translation, basically compiling to C code. Nim, Scheme (also), RPython (via PyPy, also possibly other languages it interprets, Smalltalk, JavaScript, Io, Scheme and Gameboy), Perl. Java2c. Vala. The Julia language maintains the LLVM C-Backend. Would indicate that you can take many LLVM compatible languages and output C, (So I guess you can even compile C to C? And then recursively compile that...).

How supported or performant they are will vary. And practical stuff, many of those are oddball languages. But it allows you to produce something executable for an arch that only has a C compiler.

[Cheedoong]

I will be more interested if doing it in C. Look around, most of the world's most fundamental libraries are primarily and primly written in C.

[dcbishop]

@Cheedoong: That's true, but it's more because most of them are older libraries that have been around a while than because it's a better choice. Also the OS kernels and the headers that interface with them are generally written in C.

C++, Rust and Go can all build implementations with C headers allowing them to be used as a drop in replacement.

Having said that there are a few potential problems. Most notably portability. Rust is based on LLVM so should basically work where there stuff does, but there will be older and/or less used architectures. I'm not sure of any Rust implementation for the Xtensa cpu for example (used in the ESP8266).

Also more people dealing with embedded systems and low level libraries are going to be used to C rather than the other languages.

Finally C++/Go/Rust might not end up being the general future programming language of choice.

In any case it seems to me that IPFS is still under quite a lot of design fluctuation, so it would probably be worth while holding off on a separate implementation until this are a bit more stable.

[kenCode-de]

Is there anyone working on a c-ipfs implementation yet? There seems to be a few supporters of C in here and I have 5 guys on my team here who need a c-ipfs for our firmware projects.

Please advise, because if not, then I will create the repo and would sincerely appreciate some advice and guidance with it as we progress. Please and thanx :)

[kevina]

@kenCode-de: I do not believe there is anyone working on a c-ipfs implementation.

Due the the backlash I got against a implementation in C++ (as oppose to pure C) and the fact that IPFS is rapidly changing I directed by energy elsewhere.

I will note, that if implemented right, there is no reason C++ can't be used in an resource constraint embed platform. A quick google brought up this relevant article: http://www.embedded.com/design/programming-languages-and-tools/4438660/Modern-C--in-embedded-systems---Part-1--Myth-and-Reality.

[dyba]

@kenCode-de I've been watching this discussion and this one for updates. I'm working my way through understanding basic cryptography so that I can start with a libp2p-crypto implementation in C. I identified the libp2p-crypto library as a starting point because the other libraries build on top of this. From what I've read so far, the idea I have is to use openssl to generate the RSA keys. I have no repo yet.

[kenCode-de]

On Wed, Oct 26th 2016 (next week) we begin the c-ipfs implementation. https://github.com/kenCode-de/c-ipfs

There will be 5 of us working on it to begin with. If you will be contributing, you can also join our skype group if you'd like, so just email me your skype name and I will add you. Please and Thanx! :)

cc: @jbenet

[lamarpavel]

@dyba I think your choice to start with a libp2p-crypto is right and you should create a repo for it so the ideas on how to go about doing that can be concentrated there (rather than here).

[kenCode-de]

I apologize in advance for this ping, not sure if the 8 or 9 people here who seem to be pro-C will be notified or not, just hoping you might want to join us on the C implementation next week. See my comment above, thanx guys :) @kevina @whyrusleeping @lidel @montagsoup @Kubuxu @a-shark @Cheedoong @dyba

[jmjatlanta]

I made a quick-and-dirty wrapper around the http interface using c++ and curl. It is a port of the PHP version that is available. You can find the code here: https://github.com/jmjatlanta/ipfs_curl

[vasild]

Hello, IPFS C++ API client library in the baking here: https://github.com/vasild/cpp-ipfs-api

[kenCode-de]

We have the IPFS implementation in C underway...

https://github.com/kenCode-de/c-ipfs https://github.com/kenCode-de/c-libp2p https://github.com/kenCode-de/c-multiaddr https://github.com/kenCode-de/c-multihash https://github.com/kenCode-de/c-multicodec https://github.com/kenCode-de/c-ipld https://github.com/kenCode-de/c-iprs (on hold, lack of docs)

Config file is done, Protobuff needs to be figured out, r/w storage operations in progress, implementing lightningDB (instead of leveldb), c-ipns datastore is in progress (with speed improvements), node is in progress, cid is usable now but needs a use case, also in progress: c-libp2p routing, c-libp2p crypto, c-libp2p network, is-domain, proquint, pin, key, merkledag, daemon, c-libp2p tcp, sctp, utp, udp, udt and open_tcp.

@jbenet @kevina @whyrusleeping @Kubuxu @lidel @montagsoup @a-shark @Cheedoong @dyba @PayasR

Pull Requests welcomed! :)

[PayasR]

@kenCode-de This looks like an old one! here . Glad to see the enthusiasm! :+1: I'll help!

[kenCode-de]

I have the c-ipld, c-ipfs and other repos here if you guys wanna list them: https://github.com/kenCode-de?tab=repositories

We should be done with the first formal Release by January 2nd. I have been posting my weekly updates about the different repos here: https://bitsharestalk.org/index.php/topic,22576.135.html

[jeroen]

Amazing work, looking forward to a portable ipfs library that we can link direclty into applications or high level languages.

[kenCode-de]

Thanx @jeroenooms we just posted the first Pre-Release yesterday and will have a more formal one for Release this next week too if ud like to give it a shot :)

[kenCode-de]

repo links... https://bitsharestalk.org/index.php/topic,22576.msg301280.html#msg301280

[whyrusleeping]

Hey @kenCode-de Just read through your updates (it had been a couple months since i had) and the progress is looking quite good! I like the weekly updates :+1: :ok_hand: Keep it up and we will have interop pretty soon, that will be really exciting!

[Agorise]

C-IPFS is now functional, pre-release of v1.0 is online, API and IPNS is working, just doing all the testing and fixing and testing and fixing now... Getting it to play nice with the Go version.. priceless ;) https://github.com/Agorise?tab=repositories Peace, Love, and Agorism, kenCode

[adamski]

@Agorise @kenCode-de @whyrusleeping How are things looking at the moment? This is a really promising implementation, which could in theory serve as the basis for many other platform implementations.

@whyrusleeping is there a reason why https://github.com/Agorise/c-libp2p is not listed as an official libp2p implementation?

[whyrusleeping]

@adamski we would need some docs on usage, as well as maybe an example or two (go, js and rust each have an 'echo' example) and some proof of being interoperable. I havent heard from @kenCode-de in a while, not sure how their project is chugging along

[adamski]

@whyrusleeping Yes its a shame, the documentation is very thin over there.

[whyrusleeping]

@adamski in any case, rusts FFI is compatible with C's. So you can link to and use rust-libp2p pretty easily. Does that help?

[adamski]

@whyrusleeping yes it does and something I've been looking at too. Are there any API docs for it? I posted a question there earlier: https://github.com/libp2p/rust-libp2p/issues/233

[markovchainy]

FYI I am working on a C++ implementation at https://github.com/cpp-ipfs

[Kintawesar]

@jbrooker It is thrilling! What is about https://github.com/Agorise/c-libp2p implementation? As I can see you use c++17, it is cool but won't it become a problem to interactions with another users/clients/codebases?

[markovchainy]

i don't intend to support c++98 because it's a very different language. Modern c++ is safer, faster and easier to read and write. So the only requirement is to use the c++11 ABI; c++17 code can link with c++11 code. The projects provide a dockerfile for a consistent build environment.

[Kintawesar]

Thanks for your answer. I am a c++ developer too and using modern c++. So I would be glad to join to your project and contribute

[khssnv]

Could anyone figure out what efforts are necessary to implement IPFS / pubsub for IoT devices like esp8266 / ESP32? Does anything existing on C\C++ fits to port for FreeRTOS?

[markovchainy]

Seems like there's a golang toolchain so you could work with go-ipfs and try [that].(https://github.com/andygeiss/esp32) Otherwise there is the agorise C implementation, although I'm not sure of the status of pubsub there. The C++ implementation is a long way off atm.

[Warchant]

Here is our C++ libp2p implementation https://github.com/soramitsu/kagome/tree/master/core/libp2p

[Work In Progress]

[BruceDLong]

@markovchainy How is the C++ implementation going? I would love to use it or help to finish it if possible.

[dmotte]

I am also very interested in libp2p for ESP32! I would pay for that

[autonome]

The team from libre.space is working on an IoT/embedded-focused subset of IPFS on a grant from Protocol Labs here:

https://gitlab.com/librespacefoundation/ipfs-tiny/-/wikis/final/Research-Results

[uis246]

https://github.com/mateuszviste/c-ipfs https://github.com/mateuszviste/c-libp2p

Also some here: https://git.agorise.net/agorise