RFC: Switching to C++17, with g++ 7 or higher

[sangjinhan]

Currently BEES is based on C++11 (not necessarily saying BESS is written in C++11-style😛) and built with g++5 / clang-3.8 or higher. I suggest switching to c++17, to take advantage of its new features. For example, the following code snippet:

Ethernet *eth = pkt->head_data<Ethernet *>();
Ipv4 *ip = reinterpret_cast<Ipv4 *>(eth + 1);
size_t ip_bytes = (ip->header_length) << 2;
Tcp *l4 = reinterpret_cast<uint8_t *>(ip) + ip_bytes;

could be written like this in C++17:

auto [eth, ip, l4] = pkt->Parse<Ethernet, Ipv4, Tcp>();

to reduce verbosity and hide protocol parsing details from the application logic.

For (almost) complete support for C++17, we would need g++7 or clang-4/5. These compilers may not be readily available on some systems, but since BESS has container_build.py, any systems with Docker should be able to build BESS.

Please let me know if you have some reasons that higher versions of g++ cannot be used. If I don't hear any scream for a while (a couple of weeks..?), I'll switch to C++17 and g++7 soon.

[nrdmao]

This will be pretty difficult for me to support in my current environment. I don't expect to get to g++7 in my local environment any time soon, like within the next 6 months to a year. I will look into the options with Docker, but that will also take some time as well. If you do decide to switch to g++7 please do so after a stable release point.

Thanks,

Jim

On Fri, Oct 6, 2017 at 4:42 PM, Sangjin Han notifications@github.com wrote:

Currently BEES is based on C++11 (not necessarily saying BESS is written in C++11-style😛) and built with g++5 / clang-3.8 or higher. I suggest switching to c++17, to take advantage of its new features. For example, the following code snippet:

Ethernet eth = pkt->head_data<Ethernet >(); Ipv4 ip = reinterpret_cast<Ipv4 >(eth + 1);size_t ip_bytes = (ip->header_length) << 2; Tcp l4 = reinterpret_cast<uint8_t >(ip) + ip_bytes;

could be written like this in C++17:

auto [eth, ip, l4] = pkt->Parse(Ethernet, Ipv4, Tcp);

to reduce verbosity and hide protocol parsing details from the application logic.

For (almost) complete support for C++17, we would need g++7 or clang-4/5. These compilers may not be readily available on some systems, but since BESS has container_build.py, any systems with Docker should be able to build BESS.

Please let me know if you have some reasons that higher versions of g++ cannot be used. If I don't hear any scream for a while (a couple of weeks..?), I'll switch to C++17 and g++7 soon.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/NetSys/bess/issues/668, or mute the thread https://github.com/notifications/unsubscribe-auth/AWIHrsK2jpH6nVzvqQi3jVCvu46gLuSPks5sprr-gaJpZM4PxJnG .

[sangjinhan]

Building with the Docker container should be fairly straightforward. docker pull nefelinetworks/bess_build and ./container_build.py are the only necessary steps (realized that this has never been documented though), and even OSX and Windows should be able to build BESS without requiring any software dependencies. Could you please give it a try and let me know if it works?

[nrdmao]

Sangjin,

I don't seem to be able to pull the docker image from nefellinetworks. Are you sure you have the name correct here? (Note, I am not a docker expert!)

$ docker pull nefellinetworks/bess_build Using default tag: latest Error response from daemon: repository nefellinetworks/bess_build not found: does not exist or no pull access

There are a few other issues that we will need to work through:

Our control plane is C++, not python, so I need to compile the grpc/protobuf bits in my native build environment. So I will still need to compile some of BESS natively.

As I write my own modules I will need to include them in the container images rather than in my own native environment. This complicates my build since again I need to deal with the grpc pieces as well for my private module commands and I have to manage my source files differently. Currently, I just need to keep a git archive of bess in my source tree. If I need to add stuff to this archive I will also need to manage a local git repository.

I have to think about packaging. Our software is packaged as a .swi which is a collection of RPMs. So I will need to come up with a scheme to package the bits from the docker image into an RPM.

From my build environment I can't be dependent on 3rd party sources downloaded at runtime.

Need to think about the runtime environment.

Anyway, if you are hell bent on going to g++7 I certainly can't get in your way, however when you do so, please do it after some tagged point release so that if I have to diverge from the mainline I can do so at a well defined (and tested) release point.

Thanks,

Jim

On Thu, Oct 12, 2017 at 1:58 PM, Sangjin Han notifications@github.com wrote:

Building with the Docker container should be fairly straightforward. docker pull nefellinetworks/bess_build and ./container_build.py are the only necessary steps (realized that this has never been documented though), and even OSX and Windows should be able to build BESS without requiring any software dependencies. Could you please give it a try and let me know if it works?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/NetSys/bess/issues/668#issuecomment-336264874, or mute the thread https://github.com/notifications/unsubscribe-auth/AWIHrmyMcuOsMVzewHUvWtJq8OBFDTJdks5srn1ygaJpZM4PxJnG .

[sangjinhan]

Oh sorry, it should have been nefelinetworks/bess_build (one "l" in nefeli). Updated the comment above. Please give it a try.

The container image does not include any BESS-related code, so you won't need to modify the image for your private modules. It's just a pre-configured Docker image installed with software dependencies (including protobuf and grpc) and various versions of compilers. These compilers will build files in your native environment, just like native build.

I don't know much about RPM, but presumably you only include executable and shared objects in your package, not the source code..? If so, you won't need the container image once you have the executable binary. The Docker image is just for build, not for runtime.

Maybe I am misunderstanding your concerns, but I myself have a set of private modules (with their commands) in a separate repo, which is separately managed form the main BESS trunk. Recent changes (e.g., #591, #604, ...) have made this quite easy (but not much documented yet).

I am happy to incorporate any additional features that might be necessary for out-of-tree build. :)

[nrdmao]

OK, I'll what I can do. I am not saying it is impossible, just saying that it is non-trivial. In the long run, however, it is probably a better option than my current environment especially given how quickly bess is changing as it gives me some degree of isolation from these changes that depend on the environment. However, I may not be able to work on in right away.

Thanks,

Jim

On Thu, Oct 19, 2017 at 4:04 PM, Sangjin Han notifications@github.com wrote:

Oh sorry, it should have been nefelinetworks/bess_build (one "l" in nefeli). Updated the comment above. Please give it a try.

The container image does not include any BESS-related code, so you won't need to modify the image for your private modules. It's just a pre-configured Docker image installed with software dependencies (including protobuf and grpc) and various versions of compilers. These compilers will build files in your native environment, just like native build.

I don't know much about RPM, but presumably you only include executable and shared objects in your package, not the source code..? If so, you won't need the container image once you have the executable binary. The Docker image is just for build, not for runtime.

Maybe I am misunderstanding your concerns, but I myself have a set of private modules (with their commands) in a separate repo, which is separately managed form the main BESS trunk. Recent changes (e.g., #591 https://github.com/NetSys/bess/pull/591, #604 https://github.com/NetSys/bess/pull/604, ...) have made this quite easy (but not much documented yet).

I am happy to incorporate any additional features that might be necessary for out-of-tree build. :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/NetSys/bess/issues/668#issuecomment-338061843, or mute the thread https://github.com/notifications/unsubscribe-auth/AWIHrnaYuKeeU-qP-uo9iq7ygFKOIgumks5st9V-gaJpZM4PxJnG .

[nemethf]

This might not be the right place to report but recently I had to use the container_build script, and everything worked just fine, except I could run bessctl only in the container (and bessd in the host environment). It turned out that python libraries were missing from the host. The following commands solved the issue:

~/bess$ sudo ./container_build.py shell
root@1613c41a90e2:/build/bess# cp -a /usr/local/lib/python2.7/dist-packages/{concurrent,google,grpc} .
root@1613c41a90e2:/build/bess# touch google/__init__.py
root@1613c41a90e2:/build/bess# exit
exit
Done.
~/bess$

Alternatively it might be enough to install grpcio package with pip into a virtual environment. Figuring this out wasn't straightforward to me.