[BSD] gnu sed vs bsd sed

cederom commented 1 year ago

On FreeBSD sed part of the build needs to be fixed in order to complete the sim build process :-)
Probably a gnu sed vs bsd sed issue (just like gnu make vs bsd make).
Assign to self :-)

sed: 1: "s/^\(\s\+\)\(\.init_arr ...": RE error: trailing backslash (\)

cederom commented 1 year ago

This one above is caused by:

https://github.com/apache/nuttx/blob/333707e1011ee6f17d85b455885b7b6787929b17/arch/sim/src/Makefile#L354-L358

Replacing sed with gsed fixes the issue :-)

xiaoxiang781216 commented 1 year ago

But, gsed isn't installed by default. It's better to finish the job with only POSIX defined features. BTW, it's also better to use the same approach to postpone the calling of C++ constructor in the simulator from before main to in the first user task(init). Linux rename init_array_start/init_array_end to _sinit/_einit in the link phase: https://github.com/apache/nuttx/blob/master/arch/sim/src/Makefile#L313-L325 https://github.com/apache/nuttx/blob/master/arch/sim/src/Makefile#L353-L359 macOS move the init section to the private array by a special constructor: https://github.com/apache/nuttx/blob/master/arch/sim/src/sim/posix/sim_macho_init.c

On Sun, Feb 19, 2023 at 9:52 AM Tomasz CEDRO @.***> wrote:

This one above is caused by:

https://github.com/apache/nuttx/blob/333707e1011ee6f17d85b455885b7b6787929b17/arch/sim/src/Makefile#L354-L358

Replacing sed with gsed fixes the issue :-)

— Reply to this email directly, view it on GitHub https://github.com/apache/nuttx/issues/8580#issuecomment-1435813237, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEJ24FA2VLW5PMZ5CD6MXPLWYF4FZANCNFSM6AAAAAAVATTCEI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

cederom commented 1 year ago

Long Story Short:

Linux command sed uses GNU sed, while BSD command sed uses BSD sed (GNU sed is available on BSD with gsed command), will have to find a fix for that :-)

Short Story Long:

Linux is GNU while BSD is Unix. Both are somewhat POSIX compliant ;-)
GNU Linux often introduces custom changes/extensions to its programs using the same name. For instance base shell interpreter /bin/sh is GNU Bourne-Again SHell aka bash on Linux while on Unix this is just "shell" aka /bin/sh and you need to install /usr/local/bin/bash as a separate program that is not part of the base. Writing #!/bin/bash is not the portable way.. also assuming that /bin/sh is bash is invalid (proper way is #!/usr/bin/env bash that will call bash whatever its location or #!/bin/sh if you just need sh not using anything bash specific). There are many more "enforced changes" and (self)incompatibilities in Linux world this is why I have switched to BSD for good around 15 years ago. But anyone uses what suits them best "apples and oranges"^TM :-)
There are different programs with the same name on Linux and Unix. Some are somewhat compatible, but some also have their own problems ;-)
First example is make. On Linux this is "GNU Make". On BSD this is "BSD Make". Still you can build and use "GNU Make" on BSD but the application name is gmake.
Another example is sed. On Linux this is "GNU Sed". On BSD this is "BSD Sed". Still you can build and use "GNU Sed" on BSD but the application name is gsed.
Another example is macOS that is a Mach and BSD Unix hybrid, this is why it also needs special "handling", it is closer to BSD Unix rather than Linux (you might still find some man pages or even developers from FreeBSD project working there), but it was also somehow "customized" to make Linux stuff run on it with ease (i.e. /bin/sh is /bin/bash equivalent).
Yup its a bit messy and its your guess where it came from :-P
I need to find a way to make NuttX build process as portable as possible between GNU and BSD with least possible impact on current source code :-)

If you take a look at man sed on BSD:

SED(1)                  FreeBSD General Commands Manual                 SED(1)

NAME
     sed – stream editor

SYNOPSIS
     sed [-Ealnru] command [-I extension] [-i extension] [file ...]
     sed [-Ealnru] [-e command] [-f command_file] [-I extension]
         [-i extension] [file ...]

(...)

STANDARDS
     The sed utility is expected to be a superset of the IEEE Std 1003.2
     (“POSIX.2”) specification.

     The -E, -I, -a and -i options, the special meaning of -f -, the prefixing
     “+” in the second member of an address range, as well as the “I” flag to
     the address regular expression and substitution command are non-standard
     FreeBSD extensions and may not be available on other operating systems.

HISTORY
     A sed command, written by L. E. McMahon, appeared in Version 7 AT&T UNIX.

AUTHORS
     Diomidis D. Spinellis <dds@FreeBSD.org>

BUGS
     Multibyte characters containing a byte with value 0x5C (ASCII ‘\’) may be
     incorrectly treated as line continuation characters in arguments to the
     “a”, “c” and “i” commands.  Multibyte characters cannot be used as
     delimiters with the “s” and “y” commands.

FreeBSD 13.1-RELEASE-p6          April 8, 2021         FreeBSD 13.1-RELEASE-p6

If you take a look at man gsed on FreeBSD that would be man sed on Linux :-)

SED(1)                           User Commands                          SED(1)

NAME
       sed - stream editor for filtering and transforming text

SYNOPSIS
       sed [-V] [--version] [--help] [-n] [--quiet] [--silent]
           [-l N] [--line-length=N] [-u] [--unbuffered]
           [-E] [-r] [--regexp-extended]
           [-e script] [--expression=script]
           [-f script-file] [--file=script-file]
           [script-if-no-other-script]
           [file...]

(...)

REGULAR EXPRESSIONS
       POSIX.2 BREs should be supported, but they aren't completely because of
       performance problems.  The \n sequence in a regular expression matches
       the newline character, and similarly for \a, \t, and other sequences.
       The -E option switches to using extended regular expressions instead;
       it has been supported for years by GNU sed, and is now included in
       POSIX.

BUGS
       E-mail bug reports to bug-sed@gnu.org.  Also, please include the output
       of ``sed --version'' in the body of your report if at all possible.

AUTHOR
       Written by Jay Fenlason, Tom Lord, Ken Pizzini, Paolo Bonzini, Jim
       Meyering, and Assaf Gordon.

       This sed program was built without SELinux support.

       GNU sed home page: <https://www.gnu.org/software/sed/>.  General help
       using GNU software: <https://www.gnu.org/gethelp/>.  E-mail bug reports
       to: <bug-sed@gnu.org>.

COPYRIGHT
       Copyright © 2022 Free Software Foundation, Inc.  License GPLv3+: GNU
       GPL version 3 or later <https://gnu.org/licenses/gpl.html>.
       This is free software: you are free to change and redistribute it.
       There is NO WARRANTY, to the extent permitted by law.

SEE ALSO
       awk(1), ed(1), grep(1), tr(1), perlre(1), sed.info, any of various
       books on sed, the sed FAQ
       (http://sed.sf.net/grabbag/tutorials/sedfaq.txt),
       http://sed.sf.net/grabbag/.

       The full documentation for sed is maintained as a Texinfo manual.  If
       the info and sed programs are properly installed at your site, the
       command

              info sed

       should give you access to the complete manual.

GNU sed 4.9                      December 2022                          SED(1)

acassis commented 1 year ago

@cederom in some cases it is possible to find some common parameters that will work for all systems. About the shell, normally distros based on Debian like Ubuntu uses dash instead of bash by default: https://www.baeldung.com/linux/dash-vs-bash-performance

In the past I used to remove the /bin/sh symbolic link that was pointing to dash and point it to bash to get the building system working. Nowadays I don't need to do it, everything works out-of-the-box

cederom commented 1 year ago

I think that simple SED_CMD in the top level config selected by tools/configure.sh will do the job.. just a lot of work to update all sed with SED_CMD, I hope to find some free moment for this soon :-P
This long line issue may be rewritten somehow so it works on BSD sed, but it may also reflect some bug in the bsd implementation.
FreeBSD is organized around "base system" located at / that is always common and non alterable (it can be loaded from a standard hard drive but also ROM like image of BOOTP/NFS share), and the "userland" located at /usr/local (that can be also loaded from a local hard drive but also another NFS share). All user related configurations and applications can only reside under /usr/local. The / part is nonalterable (maybe except /etc, /var, /tmp etc). This is why I cannot (don't want to) touch anything in /bin or /usr/bin because that would make my system non-compatible with the other standard FreeBSD installations. This may look a bit stiff, but this keeps the environment clean, known, and coherent for all FreeBSD installations wherever it is installed and whenever it was installed (i.e. 10 years ago or 20 years ago it was exactly the same). This is why I prefer BSD Unix over Linux.
Using GNU Sed everywhere will result in reproducible results. Even on systems that have their own sed implementation (like BSD).

I know this is quite trivial but I have some dozens urgent tasks right now. I would really dream of having more time to play with NuttX please forgive me. When I do the fix I will for sure send a PR :-)

apache / nuttx

[BSD] gnu sed vs bsd sed #8580