An experience with NixOS.

[gabriel-laddel]

I'm leaving this here in hopes that it'll be useful to someone. I didn't see anything in the NixOS manual that would indicate that I'd run into this class of problems while using it (granted, I've not read it in detail, but it's largely unreadable and overall, the whole codebase is poorly documented - attempting to discern the intent of the programmers isn't worth my time anymore, eg, where is the BNF form for the Nix language? What was so wrong with existing languages that a new one had to be invented. Where are the design documents? Perhaps I'm just not looking hard enough... in any case, looking hasn't proved useful, and reading the sources has revealed only a complete and utter disregard for quality.).

Cheers.

;;; Emacs
;;; ============================================================================
;;; First, why am I building from source when NixOS has emacs-24.4 as a package?
;;; 
;;; 1) you cannot M-. into the C sources.
;;; 2) info pages are not built by default.
;;; 3) pdf, postscript viewing via doc-view doesn't work.
;;; 
;;; The goal here is to have this (SBCL) code run without errors

(defun rp (shell-command-string)
  (uiop:run-program shell-command-string :output :string))

(defun rp-in-dir (commands dir)
  (dolist (shell-command commands)
    (rp (format nil "cd ~A && ~A" dir shell-command))))

(defun build-emacs ()
  (rp "curl http://ftp.gnu.org/gnu/emacs/emacs-24.4.tar.gz > ~/quicklisp/local-projects/emacs-24.4.tar.gz")
  (rp "cd ~/quicklisp/local-projects/ && tar zxvf emacs-24.4.tar.gz")
  (rp-in-dir '("./configure"
           "make"
           "make install")
         "~/quicklisp/local-projects/emacs-24.4/"))

;;; at the ./configure step the following error occurs,
;;; 
;; configure: error: You seem to be running X, but no X development libraries
;; were found.  You should install the relevant development files for X
;; and for the toolkit you want, such as Gtk+ or Motif.  Also make
;; sure you have development files for image handling, i.e.
;; tiff, gif, jpeg, png and xpm.
;; If you are sure you want Emacs compiled without X window support, pass
;;   --without-x
;;; 
;;; I need X support, which my current binary has via the NixOS build farms. 
;;; Let's see what they're doing to get it compile,
;;; 
;;; https://github.com/NixOS/nixpkgs/tree/master/pkgs/applications/editors/emacs-24
;;; 
;;; NOTE:
;;; regarding #3, though I'm running on X, there isn't png support in doc-view.
;;; I can open other PNG files in Emacs. I don't know the origin of this
;;; problem, it may be Emacs, whatever. I'll settle for having a working build
;;; that /should/ work and punts the problem into emacs-land.
;;; 
;;; Anyway, the relevent files,
;;; 
;;; - builder.sh: swiches out the usual C resources for Nix's C resources by
;;;               modifying Emacs's Makefiles directly.
;;;              
;;; - default.nix: the relevent bits,
;;; 
;; { stdenv, fetchurl, ncurses, x11, libXaw, libXpm, Xaw3d
;; , pkgconfig, gtk, libXft, dbus, libpng, libjpeg, libungif
;; , libtiff, librsvg, texinfo, gconf, libxml2, imagemagick, gnutls
;; , alsaLib, cairo
;; , withX ? !stdenv.isDarwin, withGTK ? true
;; }:
;; 
;; assert (libXft != null) -> libpng != null;   # probably a bug
;; assert stdenv.isDarwin -> libXaw != null;    # fails to link otherwise
;; 
;; [...]
;; 
;;   buildInputs =
;;     [ ncurses gconf libxml2 gnutls alsaLib pkgconfig texinfo ]
;;     ++ stdenv.lib.optional stdenv.isLinux dbus
;;     ++ stdenv.lib.optionals withX
;;       [ x11 libXaw Xaw3d libXpm libpng libjpeg libungif libtiff librsvg libXft
;;         imagemagick gtk gconf ]
;;     ++ stdenv.lib.optional (stdenv.isDarwin && withX) cairo;
;; 
;;   configureFlags =
;;     ( if withX && withGTK then
;;         [ "--with-x-toolkit=gtk" "--with-xft"]
;;       else (if withX then
;;         [ "--with-x-toolkit=lucid" "--with-xft" ]
;;       else
;;         [ "--with-x=no" "--with-xpm=no" "--with-jpeg=no" "--with-png=no"
;;           "--with-gif=no" "--with-tiff=no" ] ) );
;; 
;; [...]
;; 
;;; of course NixOS distributes a binary and their build farm is configured
;;; differently than mine - meaning that running "./configure
;;; --with-x-toolkit=gtk --with-xft" and "./configure --with-x-toolkit=lucid
;;; --with-xft" on my box both fail with the previously seen error.
;;; 
;;; this http://askubuntu.com/questions/375978/x11-development-libraries
;;; link indicates the neccecary codebases to build a sane Emacs,
;;; 
;;;  libpthread-stubs0 libpthread-stubs0-dev libx11-dev libx11-doc libxau-dev
;;;  libxcb1-dev libxdmcp-dev x11proto-core-dev x11proto-input-dev
;;;  x11proto-kb-dev xorg-sgml-doctools xtrans-dev
;;; 
;;; Normally I'd install them via the package manager, which I tried, but nix
;;; doesn't have anything useful in its repos.[0]
;;;
;;; Is there anyway out of this mess, satifying requirements 1-3 that doesn't
;;; force me to end up downloading and building X development libs + its 
;;; requirements etc. etc. myself?
;;; 
;;; SBCL
;;; ============================================================================
;;;
;;; I need to build SBCL-1.2.5 with threads and all the debug goodies, which
;;; corresponds to the script below (I've run this on Ubuntu previously - it
;;; works there).

(defun build-sbcl ()
  "Builds SBCL-1.2.5 from source"
  (aif (probe-file (qlpp "/sbcl"))
       (error "a SBCL repo already exists at ~a. Remove it before retrying." it)
       (rp-in-dir
    '("cd .. && git clone https://github.com/sbcl/sbcl.git"
      "git fetch origin 7db8f1a3d92c23a2459eef8fd899ac542c926d3c" ;; SBCL-1.2.5
      "git reset --hard FETCH_HEAD"
      "sh make.sh --with-sb-xref-for-internals --with-sb-threads --with-sb-qshow --with-sb-eval --with-sb-source-locations"
      "cd /doc/manual && make"
      "INSTALL_ROOT=/usr/local sh install.sh")
    (qlpp "/sbcl"))))

;;; this fails its sb-posix self-test with the output,
;;; 
;;; https://gist.githubusercontent.com/gabriel-laddel/dae75ab973af976b29c9/raw/7c5844d3d52a52c9618bf28faac5a734a3dac43f/gistfile1.txt
;;;
;;; NixOS has a script that builds SBCL from source, but it builds 1.2.0 and on
;;; further inspection they just disable the sb-posix tests.[1]
;;;
;;; I looked online and didn't see anything that would aid in fixing this.[2]
;;; 
;;; Stumpwm
;;; ============================================================================
;;;
;;; Until SBCL builds I can't compile stumpwm, but if someone has experience
;;; with this on nixos using SBCL-1.2.5 I'd be interested in knowing about it.
;;; 
;;; Conclusion
;;; ============================================================================
;;;
;;; I was asking about NixOS on the freenode channel #lisp a few days ago and
;;; it seems that they fuck with the C toolchain, breaking who knows what.[3]
;;; 
;;; The lack of the libraries I need is probably due to the difficulty of adding
;;; packages to NixOS, stemming from this arrangement. If someone knew of a hack
;;; or two to get me out of this mess and get Emacs-24.4, SBCL-1.2.5 and a
;;; recent version of Stumpwm compiling as previously specified on NixOS, I'd
;;; appreciate it.
;;;
;;; If this class of errors will emerage for as long as I use nixos, I'd like
;;; to know.
;;;
;;; Footnotes
;;; ============================================================================
;;;
;;; [0] nothing.
;;; 
;; MM> (let* ((store (uiop:run-program "ls /nix/store/" :output :string)))
;;        (filter (curry #'regex-matches "libx") (split "\\n" store)))
;; 
;; ("025ah2sy92hhn4qdjcrrvn29alr5ph81-libxkbcommon-0.3.1.tar.xz.drv"
;;  "03g8p62kmza4qx279d24mmnbawb1fvy2-libxcb-1.10.tar.bz2.drv"
;;  "116pxnak2kanp949fdvbzjh9ifxzrdl6-libxml2-2.9.1"
;;  "13pj0frvshivri4kn5n10wvqfmad5k0m-libxslt-1.1.28.drv"
;;  "1d9f8pcih74kdblqmqr0kjfhg2sly99c-libxfce4ui-4.10.0"
;;  "1qb8mpvw6bf6fc0h3lfmzky11rdky34z-libxfce4util-4.10.1.drv"
;;  "1ynkvn8yk3w2fhdynzzly9vdgyaamr21-libxfcegui4-4.10.0.drv"
;;  "20d7sf4cv3qndb64j9i3smd20vqycc2l-libxkbfile-1.0.8.drv"
;;  "2i6wskddvsgdqbjh18426ziv8xj03yn6-libxkbfile-1.0.8.drv"
;;  "2ilf5l5bn4dqc003gl32prhn97xp204i-libxklavier-5.3"
;;  "2sx34g9vm6mr3d75cph2jfb7kg6rs5aw-libxfce4ui-4.10.0"
;;  "36m59sl2y2dr1klrv47jfjq79rcgdv1a-libxml2-2.9.1.drv"
;;  "43df9c0xfdv6cwfm2jzbwqfpw2bzwlrd-libxfcegui4-4.10.0.tar.bz2.drv"
;;  "4fh3yf7v53lmdc594k676521lvz50ri9-libxml2-2.9.1"
;;  "4jspvhi9rnf7jax006zgvcalrgg9hm4g-libxcb-1.10.drv"
;;  "4l1mwx2873n7r3sm2xdw44v3lgp6hk8l-libxslt-1.1.28.drv"
;;  "4xg2qzgdvi78jkwjz7rvw66svzhkh5b8-libxklavier-5.3.tar.xz.drv"
;;  "53nnhm4c7sazhhqsiiywvcbdm8ymn8rh-libxkbfile-1.0.8"
;;  "5grpva6kminnffssy2syfmhbss6kidim-libxslt-1.1.28.tar.gz.drv"
;;  "5jlj75jcj5gq0df73yqfdzg37imyhi61-libxkbfile-1.0.8"
;;  "5jski72yiczcpa2wigiz4a41pcwhd5fa-libxfce4util-4.10.1.drv"
;;  "6d6k56c5g2p6gvwlwcj7a8n6bsc8p2gv-libxslt-1.1.28.drv"
;;  "6q17hlx5vm47rfnrf08l1rg66wjld11i-libxklavier-5.3"
;;  "6s9iiry37r1vy2k56rqh5jhl5ycr6hak-libxfce4util-4.10.1"
;;  "73bzr16i6f598jmpa1g8rln8rl5zxj2k-libxkbfile-1.0.8.tar.bz2.drv"
;;  "76vjqzbgxw61d386py8xjxjwqp2sia4n-libxklavier-5.3.tar.xz.drv"
;;  "7k0n1h5vd0iqq5s326kqwpk340pfgh2g-libxkbfile-1.0.8.tar.bz2.drv"
;;  "7ljw10rz8c4sj4rvdfgisy08q9hnv3pj-libxml2-2.9.1.drv"
;;  "86256ibvvlwfa01dixpxkbg8lhwn2xyn-libxml2-2.9.1.tar.gz.drv"
;;  "8b3f3r9qyyfpnp7srzavyc9ybfcmflkz-libxshmfence-1.1.drv"
;;  "8i705ypc81ac1v26zqc88jv3v0wk6l9y-libxml2-2.9.1.drv"
;;  "979p25zawcplfr4y1dfcaj29m1zh1hzr-libxklavier-5.3"
;;  "9smp7kdq5s4fv29dx8apbk0lci97bihk-libxcb-1.10.tar.bz2.drv"
;;  "9ybayp5y0q31520j3bj2vw6izpvcnhlc-libxcb-1.10"
;;  "a51n8sg88cvif0arnfx8yr43bzzppha4-libxml2-2.9.1.tar.gz.drv"
;;  "a8akbq6qc4mrh9kcd5akgb5nlf2p5nva-libxfcegui4-4.10.0"
;;  "ah0d8w1w75sawxb7irm1wy94q7ic4p5s-libxslt-1.1.28"
;;  "ajg296g19d9a1gap1gjfqf47yd4fhb6i-libxslt-1.1.28.tar.gz.drv"
;;  "b1w1x37caijzyn5qbaqnyaigkfs73shb-libxfce4util-4.10.1.drv"
;;  "b335b34h3xrw40p716qxkz2kpish5cxz-libxml++-2.37.1.drv"
;;  "bcqby8a58s1f6djd1n2c0xlwip72xzrl-libxcb-1.10"
;;  "bf3m978jififr7bb0zlplbprhll2l3b3-libxml2-2.9.1"
;;  "ccdkyv0igd1sc32n4n6gv5bksirdy6ib-libxkbcommon-0.4.2"
;;  "cxkx18509jylydsqcl09xnmgjy0kdj37-libxklavier-5.3.drv"
;;  "czg9czdpf9qw0kihs48j8v748lh8q42r-libxml++-2.37.1.tar.xz.drv"
;;  "d0s1yhb3vzkyhnkzhpjik52hvmr8z1dq-libxklavier-5.3.drv"
;;  "d6qpj567ldd0rmf5flw974bzn7lqdq5n-libxfcegui4-4.10.0"
;;  "d9kmn518ni0mjxymlh0zibb6fvgqiqmk-libxcb-1.10"
;;  "dy4kxm5p723wh35vpm0p022b38kdx6id-libxkbcommon-0.3.1"
;;  "fb41n9q0vyqknv9fh29h9jqq1bv46g2g-libxml2-2.9.1.tar.gz.drv"
;;  "fbf5wn8xpd9xpxbzwghs47gx4agfflf6-libxslt-1.1.28.drv"
;;  "fkmvmjchjbxr2hq1lzzj402ib7x0w478-libxshmfence-1.1"
;;  "fqbpi5d449rlj8kg5nxyvx6ldlnpvamb-libxshmfence-1.1.tar.bz2.drv"
;;  "gbjr9ajg4ncdvg8l77jwnqzny6jqjksf-libxml2-2.9.1.drv"
;;  "gzfn3syyivx8inws6pd7zia4kx2zqnw3-libxml2-2.9.1.tar.gz.drv"
;;  "h072phnwz6rvczzp2dvcwdi295mynkji-libxkbcommon-0.3.1.tar.xz.drv"
;;  "i1azqjiw6zwhhsxyh013hy93v7ijl5bd-libxml2-2.9.1.tar.gz.drv"
;;  "i990bngxygjsb7yxqmkyh4nwcwrf7lxz-libxml2-2.9.1.drv"
;;  "i9si4aqryz1506sqbkpfm87f5sxqjd7z-libxfce4ui-4.10.0.tar.bz2.drv"
;;  "iv4y7634vlpxqc5w48kifq4hn4zwm66k-libxml-1.1.7.zip.drv"
;;  "izn7kqmqnl3k4bnybjc3nyq69wrlc3yr-libxfcegui4-4.10.0.drv"
;;  "jc1qkmijvwc23sjfsc0fpfiyxvni6d8f-libxfce4util-4.10.1"
;;  "jlkj7hb36ck0p4n70lalrxfhm0krvdgr-libxfce4ui-4.10.0.tar.bz2.drv"
;;  "jzy80nbc78aa5lf4nsmr0jbj6rid12i2-libxml2-2.9.1"
;;  "k5c2wziyxs2zi2kbbki5zlmvfj84jjfa-libxkbfile-1.0.8"
;;  "kmc9vhj9rbvhs2mvyz67svsbvvxka045-libxml2-2.9.1"
;;  "krpi62lv7v0phmsld2ncjglwnv5290dq-libxcb-1.11"
;;  "kslq0annwr9qwx2zmqb5mbwcf8jsv2fj-libxkbcommon-0.4.2.drv"
;;  "kx14dfzmgf77gy9iic2n2k9ps1dhls3d-libxslt-1.1.28.tar.gz.drv"
;;  "kxm2402s1hsvaqlzj9w1wv0whf5gh79y-libxfce4util-4.10.1.tar.bz2.drv"
;;  "l6pcl98zm53vp7x93h1z043j6d6j8p5z-libxkbfile-1.0.8.tar.bz2.drv"
;;  "lhil0gadk8ys8j0v92pr27i2xcmdpp7f-libxcb-1.10.drv"
;;  "lwmgd7nfy5jnvi3zmdd3f24slcvy61j0-libxfcegui4-4.10.0.tar.bz2.drv"
;;  "m36bszgr0s0j13247ks2wjplb994im54-libxfce4ui-4.10.0.drv"
;;  "mcjn7v7kvalfxjqnzqsnpic4nhcizp7f-libxml2-2.9.1.drv"
;;  "mmhlb82z354z0whnzb3p34v397jk62ls-libxfce4util-4.10.1.tar.bz2.drv"
;;  "n5gnxw5v44p3k19crwg84c8l5dksrfpf-libxfce4util-4.10.1.tar.bz2.drv"
;;  "n6w0h59g15zqdpflc33nsyvi1yl5mrjs-libxkbfile-1.0.8.drv"
;;  "n9xij30r8z1liacqrk4zcfp1by1pj8z8-libxfcegui4-4.10.0"
;;  "ngx01x949v8lw4a5gdz7i6430klzsfix-libxslt-1.1.28.tar.gz.drv"
;;  "nilzpw1vrwhj89zm1q4xpijyz2ybkgxg-libxslt-1.1.28.tar.gz.drv"
;;  "nsrr7wgbsla23ijm31i3ny73hs3krp4i-libxfce4ui-4.10.0.drv"
;;  "pv7j4w7c98jx0qjcg10w9x5zmf9xdyqr-libxml2-2.9.1.drv"
;;  "q83lm1x1dn6s2x9skq91x2x7pwhcxn4q-libxkbcommon-0.3.1.drv"
;;  "qa6mk841n0rh060knb99qwjj2diyw0qs-libxcb-1.11.drv"
;;  "rl7xmd7c8ndw58f5n5lyh331473mzgg5-libxfce4util-4.10.1"
;;  "sg4sa9saidaxb1jnh9gzc0y07z650dk2-libxslt-1.1.28.drv"
;;  "v6nlndqwa5wqvw6q4bi0c0w09xq7bysf-libxfce4ui-4.10.0.tar.bz2.drv"
;;  "vw5dn0fqxaw6gxciilc1w0ycc88x4jsg-libxkbcommon-0.3.1.drv"
;;  "w2f5i41g0w1pyaks4icwv5rizxnfqaps-libxml2-2.9.1"
;;  "w9l2nnz3yzcpqarh26z9vzw1mv97jyq9-libxcb-1.11.tar.bz2.drv"
;;  "x0zd0ckzzgb9v8lsnnk3gahv0v2rcjnz-libxfcegui4-4.10.0.tar.bz2.drv"
;;  "x18dj0glsj8krc0la6x5q9jcd5i3awm4-libxklavier-5.3.drv"
;;  "x49n7gaj0idi9a2fjs3355fcv3blk7mg-libxklavier-5.3.tar.xz.drv"
;;  "x4aqdmrxg1k4jrnlq1pnz8d5pqgnmqii-libxkbcommon-0.4.2.tar.xz.drv"
;;  "xp4n5m67dfgcs4ybml382d3d6zfgyj4c-libxfcegui4-4.10.0.drv"
;;  "y2dwvqc8bj5iwqjgwy3pwpzlf6cm94im-libxfce4ui-4.10.0.drv"
;;  "y6r4c4nl4n6vx5nn82c6mk3k48d1kx7x-libxml2-2.9.1.drv"
;;  "z8kqkww1wqy6avcpz03c4zzqkia0jzms-libxml2-2.9.1"
;;  "zb7shk46j031c83rryjd7fyz7p1dvpif-libxcb-1.10.tar.bz2.drv"
;;  "zfgajiys8pz60ri3kpcpw55lslzn1w0b-libxml2-2.9.1.drv"
;;  "zhsl1w9lkjh44ihz27xgbxd37b89dvny-libxfce4ui-4.10.0"
;;  "zrds416mqwmx5fz3s06bvw5yc44i0b07-libxcb-1.10.drv")
;;
;;;
;;; and,
;;; 
;; MM> (filter (lambda (s) (let* ((r (scan "x[a-zA-Z0-9]*" s))) (and r (= 0 r))))
;;             (split "\\n" (nix-packages))) ;; = "nix-env --query --available"
;;
;; '("x11-ssh-askpass-1.2.4.1" "x11perf-1.5.4" "x11vnc-0.9.13" "x264-20130424_2245"
;;  "x2vnc-1.7.2" "x2x-1.27" "x86info-1.30" "xaos-3.6" "xapian-1.0.23"
;;  "xapian-1.2.8" "xapian-bindings-1.0.23" "xapian-bindings-1.2.8"
;;  "xapian-bindings-1.2.8" "xar-1.5.2" "xaralx-0.7r1785" "xarchive-0.2.8-6"
;;  "xarchiver-0.5.3" "xauth-1.0.9" "xautolock-2.2" "xawtv-3.103"
;;  "xbacklight-1.2.1" "xbase-3.1.2" "xbindkeys-1.8.6" "xbitmaps-1.1.1"
;;  "xbmc-13.2" "xbmc-13.2" "xbmc-plugin-advanced-launcher-2.5.7"
;;  "xbmc-plugin-genesis-2.1.3" "xbmc-plugin-svtplay-4.0.6" "xboard-4.7.3"
;;  "xboxdrv-0.8.5" "xburst-tools-2011-12-26" "xc3sprog-748" "xcalib-0.8"
;;  "xcape-git-2013-05-30" "xcb-proto-1.11" "xcb-util-0.3.9"
;;  "xcb-util-cursor-0.1.1" "xcb-util-image-0.3.9" "xcb-util-keysyms-0.3.9"
;;  "xcb-util-renderutil-0.3.9" "xcb-util-wm-0.4.1" "xchainkeys-0.11"
;;  "xchat-2.8.8" "xchm-1.23" "xclip-0.12" "xclock-1.0.7" "xcmiscproto-1.2.2"
;;  "xcmsdb-1.0.4" "xcompmgr-1.1.6" "xconq-7.5.0-0pre.0.20050612" "xcruiser-0.30"
;;  "xcursor-themes-1.0.4" "xcursorgen-1.0.6" "xdaliclock-2.41" "xdelta-3.0z"
;;  "xdg-user-dirs-0.15" "xdg-utils-1.1.0-rc1" "xdm-1.1.11" "xdo-0.3"
;;  "xdotool-2.20101012.3049" "xdpyinfo-1.3.1" "xdriinfo-1.0.4" "xdummy"
;;  "xerces-c-3.1.1" "xev-1.2.1" "xextproto-7.3.0" "xeyes-1.1.1"
;;  "xf86-input-evdev-2.8.4" "xf86-input-joystick-1.6.2"
;;  "xf86-input-keyboard-1.8.0" "xf86-input-mouse-1.9.1" "xf86-input-mtrack-0.3.0"
;;  "xf86-input-multitouch-20110312" "xf86-input-synaptics-1.7.6"
;;  "xf86-input-vmmouse-13.0.0" "xf86-input-void-1.4.0" "xf86-input-wacom-0.26.1"
;;  "xf86-video-ark-0.7.5" "xf86-video-ast-0.98.0" "xf86-video-ati-7.4.0"
;;  "xf86-video-cirrus-1.5.2" "xf86-video-dummy-0.3.7" "xf86-video-fbdev-0.4.4"
;;  "xf86-video-geode-2.11.16" "xf86-video-glide-1.2.2" "xf86-video-glint-1.2.8"
;;  "xf86-video-i128-1.3.6" "xf86-video-intel-2.21.15" "xf86-video-intel-2.99.912"
;;  "xf86-video-mach64-6.9.4" "xf86-video-mga-1.6.3"
;;  "xf86-video-modesetting-0.9.0" "xf86-video-neomagic-1.2.8"
;;  "xf86-video-nested-2012-06-15" "xf86-video-newport-0.2.4"
;;  "xf86-video-nouveau-1.0.10" "xf86-video-nv-2.1.20"
;;  "xf86-video-openchrome-0.3.3" "xf86-video-qxl-0.1.2" "xf86-video-r128-6.9.2"
;;  "xf86-video-savage-2.3.7" "xf86-video-siliconmotion-1.7.7"
;;  "xf86-video-sis-0.10.7" "xf86-video-suncg6-1.1.2" "xf86-video-sunffb-1.2.2"
;;  "xf86-video-tdfx-1.4.5" "xf86-video-tga-1.2.2" "xf86-video-trident-1.3.6"
;;  "xf86-video-v4l-0.2.0" "xf86-video-vesa-2.3.3" "xf86-video-vmware-13.0.2"
;;  "xf86-video-voodoo-1.2.5" "xf86-video-wsfb-0.4.0" "xf86bigfontproto-1.2.0"
;;  "xf86dgaproto-2.1" "xf86driproto-2.1.1" "xf86miscproto-0.9.3"
;;  "xf86vidmodeproto-2.3.1" "xfce4-appfinder-4.10.1" "xfce4-battery-plugin-1.0.0"
;;  "xfce4-cpufreq-plugin-1.0.0" "xfce4-datetime-plugin-0.6.1"
;;  "xfce4-dev-tools-4.10.0" "xfce4-icon-theme-4.4.3" "xfce4-mixer-4.10.0"
;;  "xfce4-notifyd-0.2.4" "xfce4-panel-4.10.1" "xfce4-power-manager-1.3.0"
;;  "xfce4-screenshooter-1.8.1" "xfce4-session-4.10.1" "xfce4-settings-4.10.1"
;;  "xfce4-systemload-plugin-1.1.1" "xfce4-taskmanager-1.0.1"
;;  "xfce4-terminal-0.6.3" "xfce4-xkb-plugin-0.5.6" "xfconf-4.10.0"
;;  "xfdesktop-4.10.2" "xfe-1.37" "xfig-3.2.5b" "xfontsel-1.0.5" "xfs-1.1.3"
;;  "xfsprogs-3.2.1" "xfwm4-4.10.1" "xgamma-1.0.5" "xhost-1.0.6" "xhtml1-20020801"
;;  "xine-lib-1.2.4" "xine-ui-0.99.7" "xineramaproto-1.2.1" "xinetd-2.3.15"
;;  "xinit-1.3.3" "xinput-1.6.1" "xinput_calibrator" "xiphos-3.2.2"
;;  "xkb-switch-1.2" "xkbcomp-1.2.4" "xkbevd-1.1.3" "xkbprint-1.0.3"
;;  "xkbutils-1.0.4" "xkeyboard-config-2.11" "xkill-1.0.4" "xlaunch"
;;  "xlibs-wrapper" "xlockmore-5.43" "xlsatoms-1.1.1" "xlsclients-1.1.3"
;;  "xlsfonts-1.0.4" "xmacro-0.3pre20000911" "xmessage-1.0.4" "xml-rpc-1.6.8"
;;  "xmlindent-0.2.17" "xmlroff-0.3.98" "xmlrpc-c-1.25.27" "xmlsec-1.2.20"
;;  "xmlstarlet-1.6.1" "xmlto-0.0.25" "xmobar-0.22.1" "xmobar-0.22.1"
;;  "xmobar-0.22.1" "xmodmap-1.0.8" "xmoto-0.5.10" "xmove-2.0b2" "xmp-4.0.7"
;;  "xmpp.py-0.5.0rc1" "xnee-3.19" "xneur-0.13.0" "xneur-0.8.0" "xonotic-0.7.0"
;;  "xorg-cf-files-1.0.5" "xorg-docs-1.7" "xorg-server-1.14.7"
;;  "xorg-sgml-doctools-1.11" "xorg-sys-opengl-3" "xorriso-1.3.8" "xoscope-2.0"
;;  "xosd-2.2.12" "xournal-0.4.8" "xpdf-3.03" "xpf-0.2" "xplanet-1.3.0"
;;  "xpr-1.0.4" "xpra-0.11.6" "xprop-1.2.2" "xproto-7.0.26" "xpwn-0.5.8git"
;;  "xrandr-1.4.3" "xrdb-1.1.0" "xrefresh-1.0.5" "xrestop-0.4" "xsane-0.999"
;;  "xscreensaver-5.30" "xsel-1.2.0" "xset-1.2.3" "xsetroot-1.1.0" "xskat-4.0"
;;  "xsokoban-3.3c" "xsynth-dssi-0.9.4" "xterm-312" "xtitle-0.2" "xtrace-1.3.1"
;;  "xtrans-1.3.4" "xulrunner-31.0" "xvfb-run" "xvidcap-1.1.7" "xvidcore-1.3.2"
;;  "xvinfo-1.1.2" "xwd-1.0.6" "xwininfo-1.1.3" "xwud-1.0.4" "xxdiff-4.0"
;;  "xylib-1.3" "xz-5.0.5")
;;;
;;; [1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/sbcl/1.2.0.nix
;;; 
;;;     I don't know sed but ran the scripts on my local SBCL dir and checked
;;;     the git changeset - they just remove the source for the tests
;;;     completely. The relevent bits from the .nix file:
;;;
;;   # SBCL checks whether files are up-to-date in many places..
;;   # Unfortunately, same timestamp is not good enough
;;   sed -e 's@> x y@>= x y@' -i contrib/sb-aclrepl/repl.lisp
;;   sed -e '/(date)/i((= date 2208988801) 2208988800)' -i contrib/asdf/asdf.lisp
;;   sed -i src/cold/slam.lisp -e \
;;     '/file-write-date input/a)'
;;   sed -i src/cold/slam.lisp -e \
;;     '/file-write-date output/i(or (and (= 2208988801 (file-write-date output)) (= 2208988801 (file-write-date input)))'
;;   sed -i src/code/target-load.lisp -e \
;;     '/date defaulted-fasl/a)'
;;   sed -i src/code/target-load.lisp -e \
;;     '/date defaulted-source/i(or (and (= 2208988801 (file-write-date defaulted-source-truename)) (= 2208988801 (file-write-date defaulted-fasl-truename)))'
;; 
;;   # Fix the tests [editor's note: 'fix' the tests by removing them? fuck you.]
;;   sed -e '/deftest pwent/inil' -i contrib/sb-posix/posix-tests.lisp
;;   sed -e '/deftest grent/inil' -i contrib/sb-posix/posix-tests.lisp
;;   sed -e '/deftest .*ent.non-existing/,+5d' -i contrib/sb-posix/posix-tests.lisp
;;   sed -e '/deftest \(pw\|gr\)ent/,+3d' -i contrib/sb-posix/posix-tests.lisp
;;   sed -e '5,$d' -i contrib/sb-bsd-sockets/tests.lisp
;;   sed -e '5,$d' -i contrib/sb-simple-streams/*test*.lisp
;;;
;;; [2] none of the following turn up anything promising in the first 10 results
;;;
;; (dolist (q '("WARNING! Some of the contrib modules did not build successfully or pass their self-tests. Failed contribs: sb-posix"
;;       "nixos sbcl issues"
;;       "nixos sbcl posix issues"
;;       "sbcl 1.2.5 posix linux"
;;       "sbcl posix test fail on nixos"
;;       "sbcl 1.2.5 posix linux tests fail"))
;;   (google q))
;;;
;;; [3] ...
;;;
;; 10:08 <gabriel_laddel> I'm working on a NixOS box I've been setting up as a
;;                        CL-only environment. Downloading iolib via quicklisp
;;                        currently throws an error, as the groveler can't
;;                        libfixposix. I installed libfixposix, restarted the
;;                        build and it still throws. I've no understanding of
;;                        g++ (the external process called by the iolib
;;                        groveler) and would like to maintain my innocence if
;;                        at all possible. Is anyone familiar with iolib & g++
;;                        and know
;; 10:08 <gabriel_laddel> s of a quick hack to point the groveler it at the
;;                        correct sources? Anything goes really - copying the
;;                        sources of libfixposix from A to B, monkey patching
;;                        iolib etc.
;; 10:09 <fe[nl]ix> gabriel_laddel: what do you mean you installed libfixposix ?
;; 10:11 <gabriel_laddel> fe[nl]ix: I downloaded it using the nixos package
;;                        manager. I should note that it 'just works' on ubuntu
;;                        when you do the ~same.
;; 10:12 <fe[nl]ix> gabriel_laddel: I don't think that iolib will work easily on
;;                  nixos
;; 10:12 <fe[nl]ix> or any cffi library, for that matter
;; 10:13 <gabriel_laddel> fe[nl]ix: care to explain why? I've no (blessedly)
;;                        experience with C/C++ toolchains.
;; 10:14 <fe[nl]ix> gabriel_laddel: AFAIK, nixos deals with multiple
;;                  installations of the same library by hard-coding the full
;;                  paths in the depending binaries
;; 10:14 <fe[nl]ix> using ld's rpath
;; 10:15 <fe[nl]ix> whereas CFFI basically uses dlopen() wuth just the library's
;;                  soname hoping that the linker will find it in some global
;;                  namespace

[gabriel-laddel]

I tried building SBCL-1.2.5, cutting out the SB-POSIX tests until I can figure out something better.

This is SBCL 1.2.5.53-c787e53-dirty, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
fatal error encountered in SBCL pid 30324(tid 140737354086144):
can't load .core for different runtime, sorry

Welcome to LDB, a low-level debugger for the Lisp runtime environment.
ldb>

I'm betting that this is NixOS related. there were issues trying to run an XULrunner binary too for whatever it's worth.

[thoughtpolice]

Some notes:

1) For the overall design of the language and concepts, Eelco's papers are (for better or worse) the best place to look, I think. They describe the language, its motivations, and NixOS rather thoroughly. Some good starting points are Nix: A Safe and Policy Free System for Software Development, and NixOS: A Purely functional Linux distribution.

It's been known for a while that unfortunately documentation is lacking on a few of these fronts. It's absolutely something we need to fix (I too think the NixOS on-boarding and beginner material is fairly sub-par).

2) Related to the above, there is no true BNF describing the Nix parser, unfortunately. This was a question raised recently in an attempt to write another Nix implementation in Haskell. A proper BNF is something that should be easy to add to the documentation somewhere.

3) I am confused by your inquiry about libraries being available. libx11, libxau, etc are all readily available in Nixpkgs (if they weren't, many other things would not be either). Your query of ls /nix/store does not tell you anything useful or meaningful. You must query the actual repository, which is described using Nix. For example:

$ nix-env -qa | egrep -i '(libx11|libxau)'
libX11-1.6.2
libXau-1.0.8

In your paste of the Emacs default.nix file, one dependency is specified as x11. This is not the same as libX11 - indeed, x11 is actually a meta-package which encompasses the modular Xorg framework and many libraries, including libXext, libXft, libX11, etc etc. You may determine this by looking at what the name x11 is bound to in pkgs/top-level/all-packages.nix and seeing what other packages it incorporates.

That said, I'm afraid I cannot help you with SBCL or StumpWM. I use neither. As it stands today, the NixOS community is structured such that all Nix users are, in essence, de-facto Nix developers. This community is still so small it is almost a requirement for you to use it regularly. In return, most of us believe the benefits are massive enough to deal with that. But it is not true of everyone. That is an understandably painful reality, but it's a reality we cannot yet change. Hopefully in the future we can have enough users and dedicated maintainers to alleviate this burden from the userbase, and StumpWM and SBCL and X number of other things will just work for anyone who wants them, with no effort.

If a package is broken or outdated or unmaintained (which happens with any distro), it almost always falls to a user who wishes to use that software to maintain it, I'm afraid. Have you tried asking on the #nixos channel on freenode? I've found many helpful people there in my time who may be willing to assist you in getting a more recent SBCL and StumpWM working.

---

Oh, and one final thing about your 'editors note' (which I'll quote to ensure everyone can read it for the future):

# Fix the tests [editor's note: 'fix' the tests by removing them? fuck you.]

If you wish to behave this way, I'm afraid my impression is that your mind about how to conduct yourself in public spaces is fairly made up - and no amount of honest actors will change that behavior or attitude.

I can understand frustration at using new software and being perplexed by it. But your tone in here is pretty clear. And I imagine those aren't attitudes or tone we want anyway - besides, nothing brought you here beyond your own desires.

So, if that is the case, maybe it would be best for you to find a community that does appreciate that sort of attitude - perhaps the Common Lisp community does. Or the Xbox Live community, perhaps.

[gabriel-laddel]

1) For the overall design of the language and concepts, Eelco's papers are (for
better or worse) the best place to look, I think. They describe the language,
its motivations, and NixOS rather thoroughly. Some good starting points are Nix:
A Safe and Policy Free System for Software Development, and NixOS: A Purely
functional Linux distribution.

For worse.

A proper BNF is something that should be /easy/ to add to the documentation somewhere.

I would use the term "tedious bullshit" to describe this task, but whatever, moving on.

3) I am confused by your inquiry about libraries being available. libx11,
libxau, etc are all readily available in Nixpkgs (if they weren't, many other
things would not be either). Your query of ls /nix/store does not tell you
anything useful or meaningful. You must query the actual repository, which is
described using Nix. For example:

$ nix-env -qa | egrep -i '(libx11|libxau)'
libX11-1.6.2
libXau-1.0.8

I'm an idiot, thank you.

In your paste of the Emacs default.nix file, one dependency is specified as
x11. This is not the same as libX11 - indeed, x11 is actually a meta-package
which encompasses the modular Xorg framework and many libraries, including
libXext, libXft, libX11, etc etc. You may determine this by looking at what the
name x11 is bound to in pkgs/top-level/all-packages.nix and seeing what other
packages it incorporates.

I'll look into this now.

That said, I'm afraid I cannot help you with SBCL or StumpWM. I use neither. As
it stands today, the NixOS community is structured such that all Nix users are,
in essence, de-facto Nix developers. This community is still so small it is
almost a requirement for you to use it regularly. In return, most of us believe
the benefits are massive enough to deal with that. But it is not true of
everyone. That is an understandably painful reality, but it's a reality we
cannot yet change. Hopefully in the future we can have enough users and
dedicated maintainers to alleviate this burden from the userbase, and StumpWM
and SBCL and X number of other things will just work for anyone who wants them,
with no effort. If a package is broken or outdated or unmaintained (which
happens with any distro), it almost always falls to a user who wishes to use
that software to maintain it, I'm afraid.

WIP, K.

Have you tried asking on the #nixos channel on freenode? I've found many helpful
people there in my time who may be willing to assist you in getting a more
recent SBCL and StumpWM working.

curl http://nixos.org/irc/logs/log.20141112 | grep gabriel_laddel

Oh, and one final thing about your 'editors note' (which I'll quote to ensure
everyone can read it for the future):

# Fix the tests [editor's note: 'fix' the tests by removing them? fuck you.]

If you wish to behave this way, I'm afraid my impression is that your mind about
how to conduct yourself in public spaces is fairly made up - and no amount of
honest actors will change that behavior or attitude.

(dolist (k '("define remove" "define fix" "define test") (google k)))

I can understand frustration at using new software and being perplexed by it.

(google "define perplexed") ;=> 

per·plexed/pərˈplekst/
adjective

    completely baffled; very puzzled.

per·plex/pərˈpleks/
verb

    (of something complicated or unaccountable) cause (someone) to feel
    completely baffled.

There is nothing perplexing about Nix.

But your tone in here is pretty clear. And I imagine those aren't attitudes we
want anyway

A person with the handle thoughtpolice doesn't want anyone to use mean words or to be critical of a project he is involved with? Color me shocked.

besides, nothing brought you here beyond your own desires.

...

So, if that is the case, you should find a community that does appreciate that
sort of attitude - perhaps the Common Lisp community does. Or the Xbox Live
community, perhaps.

sizzle

Thanks for the help.

[thoughtpolice]

A person with the handle thoughtpolice doesn't want anyone to use mean words or to be critical of a project he is involved with?

No, actually. It's because I'm a developer who works on other open source projects (several for my job), and realistically I have better things to do than deal with annoying people telling me and other people in the projects I work on to go fuck themselves - which substantially decreases my motivation to deal with said project in the first place. Indeed, my respect lies with the NixOS developers, and not you. So you'll have to forgive me if I find you annoying, but that's life sometimes.

This should be pretty plainly obvious (as opposed to your quite rude assumption of my motivations and presumably willful misinterpretation of it, as if it would help your position). But apparently it needs to be spelled out to some people who are seemingly incapable of understanding these aspects of social interaction and why their behaviors might be considered harmful. Which I believe are pretty clearly alluded to in my original posting.

Of course, I'm not about to succumb to such belittling tactics by people like you - that would only prove such tactics work to drive people away, ultimately harming everyone. So I'm afraid the feared Orwellian Thought Police of NixOS™ is here to stay -- much to your great despair, I imagine. Be afraid - be very afraid!

---

That said, this bug is a bit convoluted. Really there are several things going on here, based on your original description:

1) The emacs package is missing several things, including:

• Infopages are not built by default.

• PDF/postscript viewing doesn't work.

  2) Dependencies for Emacs are unclear, although as I explained before, we should have all of these readily available. They merely follow different conventions for naming/meta-packaging. If something is missing, it can certainly be upstreamed.

  3) SBCL tests should not be nerfed; furthermore they should likely be integrated into the build process on Hydra, so ever. How/if this can be fixed depends on the upstream project and the tests in question; while many projects do enable tests (by specifying doCheck = true; in their default.nix), this isn't always easily doable in some cases. I'm afraid I don't know much about the specific tests in question here to comment.

  4) SBCL dynamic dependencies/cffi libraries need to be properly managed in some way. There are a slew of other packages that manage their own subpackages in a similar way; Emacs is one of them, in fact. Another is OCaml or the Haskell package set. In short, we tend to write environment wrappers which properly 'set up' the environment for a specific package.

  For example, when we use GCC and specify a library like libxml2 in the buildInputs of an executable, this actually is translated into a set of paths in which GCC will look for libraries - this way, #include directives properly work, as well as flags like -lxml2. These paths are actually provided via environment variables - and GCC is actually a shell script that wraps the real GCC in with the proper flags.

  This explains why we have packages like clang-wrapper or whatnot. We would probably equally need something like an sbcl-wrapper tool, that when invoked will properly set up an environment in which the real SBCL executable will be able to dlopen() the right libraries (assuming of course there's some flag/option to control the base directory of where the dlopen occurs and where SBCL searches for shared objects in the first place).

  Then, individual SBCL packages would go into their own namespace, such as nixpkgs.sbclPackages, where people could select the sub libraries they want. We have quite a few toolchains that follow this exact scenario.

  5) StumpWM and SBCL are outdated it seems, and could do with a working update.

Really though, these are all separate issues in a way, so I would suggest you file them each in a ticket so they can be triaged and more easily tracked by developers.

Having a meta-ticket is fine in general AFAIK (maybe 'overhaul SBCL support' which most of the issues would fall under), although keeping separate issues for the separate subtasks makes it easier to manage them and address individual concerns.

Of course, I'm not going to do this for you, but I'd suggest that it's probably the easiest way to at least have your complaints looked at (as opposed to posting a very long Emacs buffer - the syntax highlighting actually makes it rather painful to read).

Thanks for the help.

Don't worry about it - although you won't be getting any from me in the future. Although I will certainly return and be sure to call you out on shitty behavior should I see it. After all, what kind of Thought Police™ would I be if I didn't? I'd be letting you down, what with the expectations you set for me!

[wmertens]

This issue is fascinating, I've never seen a bug report written in LISP before. I'm also a little perplexed by the general tone and sense of entitlement but anyway... I'd like to address some of your points @gabriel-laddel .

I see that you try to build EMACS from the command line. That's not how it is done in NixOS as that's not repeatable and stateful. Either you clone nixpkgs and change the nix expressions already there, or you use the packageOverrides in your ~/.nixpkgs/config.nix (see http://lethalman.blogspot.it/2014/11/nix-pill-17-nixpkgs-overriding-packages.html).

You asked 1 question on #nixos and then logged off when you got no replies? Perhaps the nix-dev mailing list would be more suited to your IRC style.

I general I would say there is a passion for code quality and statelessness in NixOS, just go look at all the comments on issues asking to change this or that. You seem to have gotten the reverse impression from the learning curve and EMACS. Nothing I can do about that, just interesting. If you like the Nix concepts you could implement StumpWM the way you like it, or you could just walk away.

I don't know what causes your SBCL error. The guys in the LISP channel are correct, when you build something for Nix you fix all inputs, and if something doesn't work like that you wrap it so it only sees the part of the world that you want.

Oh, those sb-posix test failures are also perplexing. Expecting 13, getting /? Wow. Of course, just turning them off was not the right thing to do.

Anyway.

[wmertens]

Oh and hats off to @thoughtpolice for his detailed and gracious answers.

[thoughtpolice]

Oh, and I will leave one mention about the IRC channel: I am not sure in what timezone the OP resides, but realistically from my time contributing I've noticed NixOS has a very large European userbase and a much smaller "not European" userbase. Given the assumption the logs are timestamped to a European timezone, it's perhaps not surprising most people weren't available in the early morning hours.

So knowing nothing about OP, I imagine this could possibly have some impact on the availability of people to respond. As an American myself, this too has caught me off guard. So yes, the nix-dev mailing list may actually be a better place to formulate longer discussions, depending on your availability.

[7c6f434c]

Also, currently NixPkgs has a working StumpWM 0.9.8 package (previous release) compiled using SBCL-1.2.5 (which is also in NixPkgs).

Some of SBCL tests make assumptions about the base system; the others are run during the SBCL build: http://hydra.nixos.org/build/16738664/log/raw (look for ACLREPL-TESTS)

[Shados]

@gabriel-laddel:

eg, where is the BNF form for the Nix language?

There's an SDF representation of the syntax in @edolstra's PHD thesis starting on page 64, though I'm not sure if that would be entirely up to date. It also can be seen as acting as a design document, and lays out the thought and central metaphors behind Nix/OS pretty clearly.

[wmertens]

@Shados actually, you could say that https://github.com/NixOS/nix/blob/b6809608cc467925db44b1eb435095c37e433255/src/libexpr/parser.y#L298-L523 (Bison configuration) is a very strict, machine-readable BNF form... I wonder if there's a tool that takes Bison input and generates a human-readable grammar.

[gabriel-laddel]

I've seen everyone's responses and will be responding in due time.

[wmertens]

Relevant, apparently you can get an almost-BNF out of bison with -v: http://stackoverflow.com/questions/19477103/extract-bnf-grammar-rules-from-yacc-file

[wmertens]

FWIW, here's the grammar. The tokens are defined in https://github.com/NixOS/nix/blob/master/src/libexpr/lexer.l#L80.

0 $accept: start $end

1 start: expr

2 expr: expr_function

3 expr_function: ID ':' expr_function
4              | '{' formals '}' ':' expr_function
5              | '{' formals '}' '@' ID ':' expr_function
6              | ID '@' '{' formals '}' ':' expr_function
7              | ASSERT expr ';' expr_function
8              | WITH expr ';' expr_function
9              | LET binds IN expr_function
10              | expr_if

11 expr_if: IF expr THEN expr ELSE expr
12        | expr_op

13 expr_op: '!' expr_op
14        | '-' expr_op
15        | expr_op EQ expr_op
16        | expr_op NEQ expr_op
17        | expr_op '<' expr_op
18        | expr_op LEQ expr_op
19        | expr_op '>' expr_op
20        | expr_op GEQ expr_op
21        | expr_op AND expr_op
22        | expr_op OR expr_op
23        | expr_op IMPL expr_op
24        | expr_op UPDATE expr_op
25        | expr_op '?' attrpath
26        | expr_op '+' expr_op
27        | expr_op '-' expr_op
28        | expr_op '*' expr_op
29        | expr_op '/' expr_op
30        | expr_op CONCAT expr_op
31        | expr_app

32 expr_app: expr_app expr_select
33         | expr_select

34 expr_select: expr_simple '.' attrpath
35            | expr_simple '.' attrpath OR_KW expr_select
36            | expr_simple OR_KW
37            | expr_simple

38 expr_simple: ID
39            | INT
40            | '"' string_parts '"'
41            | IND_STRING_OPEN ind_string_parts IND_STRING_CLOSE
42            | PATH
43            | SPATH
44            | URI
45            | '(' expr ')'
46            | LET '{' binds '}'
47            | REC '{' binds '}'
48            | '{' binds '}'
49            | '[' expr_list ']'

50 string_parts: STR
51             | string_parts_interpolated
52             | %empty

53 string_parts_interpolated: string_parts_interpolated STR
54                          | string_parts_interpolated DOLLAR_CURLY expr '}'
55                          | STR DOLLAR_CURLY expr '}'
56                          | DOLLAR_CURLY expr '}'

57 ind_string_parts: ind_string_parts IND_STR
58                 | ind_string_parts DOLLAR_CURLY expr '}'
59                 | %empty

60 binds: binds attrpath '=' expr ';'
61      | binds INHERIT attrs ';'
62      | binds INHERIT '(' expr ')' attrs ';'
63      | %empty

64 attrs: attrs attr
65      | attrs string_attr
66      | %empty

67 attrpath: attrpath '.' attr
68         | attrpath '.' string_attr
69         | attr
70         | string_attr

71 attr: ID
72     | OR_KW

73 string_attr: '"' string_parts '"'
74            | DOLLAR_CURLY expr '}'

75 expr_list: expr_list expr_select
76          | %empty

77 formals: formal ',' formals
78        | formal
79        | %empty
80        | ELLIPSIS

81 formal: ID
82       | ID '?' expr

[peti]

There has been no activity for a couple of weeks, so I'll close the issue. If there are actionable items hidden in this thread somewhere, I guess it would be better to create separate, new tickets for those.

[gabriel-laddel]

I'll note that I've drafted my response to all unanswered messages but am busy and don't know when I'll have time to finish it off.

Closing the issue is fine with me.

[wmertens]

@gabriel-laddel just paste your draft? Otherwise you'll never come back to it...

[gabriel-laddel]

I don't publish drafts. I'm in the middle of something at the moment and will respond over the weekend.

[vcunat]

:-) reminds me of classical Cathedral vs. Bazaar dilemma.

[gabriel-laddel]

Wasn't able to finish over the weekend and am busy for the next two days. Best guess is wed/thurs.

[gabriel-laddel]

Note1: I've read NixOS: A Purely Functional Linux Distribution and Nix: A Safe and Policy-Free System for Software Devlopment, but not the associated PHD thesis. The contents listing didn't indicate that it'd introduce me to any new ideas.

Note2: I've ignored forth and apl derivatives in this comment. Anyone familiar with will understand why upon reading it.

Note3: I posted and deleted this approximately ~5min ago - sorry, the using ``` as blockquote didn't work as expected.

Fundamentally, package management is composed of graph traversals and transformations. The goal that separates NixOS from other package managers, reproducible builds, can be rendered as "the ability to save and restore unix's dependency graph". This is a neat idea, but the implementation merely trades one set of problems for another. Many crucial design decisions that should have been throughly discussed were completely ignored. For example, nowhere in the design documents is it stated that every(?) programming language already has its own package manager(s) with its own idiosyncrasies that for the most part don't share concepts or code with the others. There are many good questions that originate from this fact, were one to acknowledge it. Examples: at what point is appropriate to leave package management up to a language's ecosystem? Is there ever a reason to? Can maintainers be convinced that the 'nixos way' is a good one and that packaging their code with a flag of some sort that produces a nix friendly build script is a reasonable request? Are we confident enough in nix to spend their time? Given that the NixOS plan is to modify build scripts and sources to its liking, what is the proper way to go about doing source-to-source transformations? Are there commercial offerings that solve this problem? The undiscussion of these issues is prototypical of the Nix design documents and documentation.

From NixOS: A Purely Functional Linux Distribution,

Runtime dependencies In Nix, we generally try to fix runtime dependencies at build time. This means that while a program may execute other programs or load dynamic libraries at runtime, the paths to those dependencies are hard-coded into the program at build time. For instance, for ELF executables, we set the RPATH in the executable such that it will find a statically determined set of library dependencies at runtime, rather than using a dynamic mechanism such as the LD_LIBRARY_PATH environment variable to look up libraries. This is important, because the use of such dynamic mechanisms makes it harder to run applications with conflicting dependencies at the same time (e.g., we might need Firefox linked against GTK 2.8 and Thunder bird linked against GTK 2.10). It also enhances determinism: a program will not suddenly behave differently on another system or under another user account because environment variables happen to be different. However, there is one case in NixOS and Nixpkgs of a library dependency that /must/ be overridable at runtime and cannot be fixed statically: the implementation of OpenGL to be used at runtime (libGL.so) which is hardware specific. We build applications that need OpenGL against Mesa, but add the impure (stateful) path /var/run/opengl-driver to the RPATH. The activate script syslinks that path to the actual OpenGL implementation selected by the configuration (e.g., nvidiaDriver) to allow programs to use it.

[...]

Build actions The Nix /model/ is that derivations are pure, that is, two builds of identical derivation should produce the same result in the Nix store. However, in contemporary operating systems, there is no way to actually enforce this model. Builders can use any impure source of information to produce the output, such as the system time, data downloaded from the network, or the current number of processes in the system as seen in /proc. It is trivial to construct a contrived builder that does such things. But build processes in the system as seen in /proc. But build processes generally do not, and instead are fairly determinisitc; impure influences such as the system time generally do not affect the runtime behavior of the package in question.

There are however frequent exceptions. First, many build processes are greatly affected by environment variables, such as PATH or CFLAGS. Therefore we clear the environment before starting a build (except for the attributes variable to a non-existent directory, because some derivations (such as Qt) try and read settings) from the user's home directory.

Second, almost all packages look for dependencies in impure locations such as /usr/bin and /usr/include. Indeed, the undeclared dependencies caused by this behavior are what motivated Nix in the first place: by storing packages in isolation from each other, we prevent undeclared build-time dependencies. In five years we haven't had a single instance of a package having an undeclared build-time dependency on another package /in the Nix store/, or having a runtime dependency in the Nix store not detected by the reference scanner. However, with Nix under other Linux distributions or operating systems, there have been numerous instances of packages affected by paths outside the Nix store. We prevent most of those impurities through a wrapper script around GCC and ld that ignores or fails on paths outside of the store. However, this cannot prevent undeclared dependencies such as direct calls to other programs, e.g., a Makefile running /usr/bin/yacc.

Since NixOS has no /bin, /usr and /lib, the effect of such impurities is greatly reduced. However, even in NixOS such impurities can occur. For instance, we recently encountered a problem with the build of the dbus package, which failed when /var/run/dbus didn't exist.

As a final example of impurity, some packages try to install files under a different location than $out. Nix causes such packages to /fail deterministically/ by executing builders under unprivileged UIDs that do not have write permisions to other store paths other than $out, let alone paths such as /bin. This packages must then be patched to make them well-behaved.

To ascertain how well these measures work in preventing impurities in NixOS, we performed two builds of the Nixpkgs collection on two different NixOS machines. This consisted of building 485 non-fetch url derivations. The output consisted of 165927 files and directories. Of these, there was only one /file name/ that differed between the two builds, namely in mono-1.1.4: a directory gac/IBM.Data.DB2/1.0.3008.37160_7c307b91aa13d208 versus 1.0.3008.40191_7c307b91aa13d280. The differing number is likely derived from system time.

We then compared each file. There we differences in 5059 files, or 3.4% of all regular files. We inspected the nature of the differences: almost all were caused by timestamps being encoded in files, such as in Unix object files archives or compiled Python code. 1048 Emacs Lisp files differed because the hostname of the build machines were stored in the output. Filtering out these and other files types that are known to contain timestamps as well (such as a build process inserting the build time in a C string). This hypothesis is strongly0 supported by the fact that of those, only 42 (or 0.03%) had different files sizes. None of these content differences have ever caused an observable difference in behavior.

How exactly does the reference scanner detect runtime dependencies? Regex and sed hackery? Or the correct way, parsing each language into its abstract syntax tree (ast), walking that and finding references to unix level dependencies (both static and dynamic)? I assume the former, as the latter is a great deal of work and would have been mentioned. Also, the only complete C99 parser (i.e., one that takes into account all the GCC extensions used in the linux kernel) I'm aware of is the haskell package Language.C.AST - and nix doesn't use haskell.

The problem with anything other than traversing a language's ast is that you're forever stuck with an approximation of the input language, resulting in false positives etc. This is not new information.

Much of the New Jersey approach is about getting away with less than is necessary to get the /complete/ job done. E.g., perl, is all about doing as little as possible that can approximate the full solution, sort of the entertainment industry's special effects and make-believe works, which for all practical purposes /is/ the real thing. Regular expressions is a pretty good approximation to actually parsing the implicit language of the input, too, but the rub with all these 90% solutions is that you have /no/ idea when they return the wrong value because the approximation destroys any ability to determine correctness. Most of the time, however, the error is large enough to cause a crash of some sort, but there is no way to do transactions, either, so a crash usually causes a debugging and rescue session to recover the state prior to the crash. This is deemed acceptable in the New Jersery approach. The reason they think this also /should/ be acceptable is that they believe that getting it exactly right is more expensive than fixing things after crashes. Therefore, the whole language must be optimized for getting the first approximations run fast.

-- http://www.xach.com/naggum/articles/3244575386963745@naggum.no.html

Notice that the paper says,

by storing packages in isolation from each other, we prevent undeclared build-time dependencies. In five years we haven't had a single instance of a package having an undeclared build-time dependency on another package /in the Nix store/, or having a runtime dependency in the Nix store not detected by the reference scanner.

but fails to mention how many people used NixOS over those 5 years, how many times undeclared build time dependencies /outside/ of the Nix store were encountered or how many packages were in the Nix store over this time period. Is this not relevant information? As Naggum stated above: "the rub with these 90% solutions is that you have /no/ idea when they return the wrong value becuase the approximation destroys any ability to determine correctness." If say, most of the NixOS users are NixOS developers (which they are) and the reference scanner doesn't walk the ast of the programs in question (which it doesn't), no reported failures means exactly nothing, because there will be many code paths not encountered by its users due to the sheer complexity of the software being interfaced with (e.g., how many NixOS +developers+ users use emacs docview? Not many apparently).

There are other issues with these papers I'm going to skip over, such as the clearing of environment variables (the proper thing to do imo is to gensym them to prevent unwanted interactions and study how they're being used - filter for 'functional' builds...), failure to discuss why a new project is needed in the first place (why can't you just perform this computation via an extension to an existing project?), total wtf issues with the papers (who are they written for? One who needs the concept of "file system" explained is going to have nfi what emacs lisp or ~/what/all/the/slashes/signify) etc.

The heart of the matter is that unix (ostensibly[0]) plays host to many different philosophies of how to go about programming. Languages differ in their approach to build systems, package management, syntax and foreign function interfaces. To change the way many software packages are built in a semi-automated, non-ast-walking fashion is insane. Not to discuss this in the design documents is insane. NixOS has 46 contributors and the documentation is sufficiently general as to give each contributor his own wildly different interpretation of the scope of the project. As far as I can tell, were one to extrapolate from the given information to a set of concrete requirements we see that NixOS plans to rewrite the build scripts for every version of every project on unix. Again, this is insane. The correct thing to do in this situation is to realize the utter impossibility of the task that has been set forth and re-evaluate one's approach.[1]

Let's examine the reference scanner. What is the correct way to approach the problem? We know regular expressions cannot respect the semantics of a language and thus are (unless used as a part of a larger parser) inappropriate for meta-programming (source-to-source transformations). Thus, we must work by manipulating the ast of various languages, and then pretty printing the altered code into the language's concrete syntax.

Most languages don't offer the ability to do this, and if one examines e.g., Clang, he will see all sorts of nonsense about concrete/surface syntax, context free grammars and compiler technologies. This certainly can't all be necessary (hint: it isn't).

Consider the following mathematical expression:

(3 + 2) * 8 / 3 * 3^6

Fully parenthesizing yields:

(((3 + 2) * 8) / (3 * 3^6))

When computers execute programs, or humans mathematics, the order of operations must be taken into account. one way to notate the previous expression to remove such ambiguities present in mathematical notation is to move operators to the front of each parenthesized list, passing the remaining elements as its arguments. This is known as fully-parenthesized prefix notation.

(/ (* (+ 3 2) 8) (* 3 (^ 3 6)))

This notational scheme has a direct mapping to the program's ast, which can be rendered as

http://i.imgur.com/8TO5VcK.png

See the pattern? An s-expression (fully parenthesized prefix notation) is merely a serialization for the ast. Thus, manipulations of the sexpr correspond to manipulations of the ast. Languages with the vocabulary to manipulate the s-expression structure can manipulate themselves (their ast) just as easily as any other thusly structured information. It follows from this that all software 'tooling' and translation tasks, are transparently tree traversals e.g., find me all things that call this function, reference this variable, modify all build scripts to take into account the new &optional argument in the updated library, etc.[2]

For whatever reason, many find s-expressions distasteful and spend much of their lives attempting to add the same meta-programming facilities to ALGOL derived languages. They all have failed. Not because of any sort of mechanical failing of the computer, but the human's inability to fully comprehend the parsing and syntactical schemes they're able to create (Ruby's parser is 10k lines of C, Clang's is >100k loc. Note that the entirety of SBCL is ~300k loc, and has much fat that could be trimmed off - http://www.cliki.net/Lisp%20-%20Next%20Generation).

Scala is a notable failure in this regard. Watch this video:

http://www.youtube.com/watch?v=TS1lpKBMkgg

Pay attention to 37:39-42:50 and you'll get to see Paul Phillips flipping out over ir/asts (same thing!). He even states his plan for the next 25 years - attempt to solve a problem solved 50+ years ago (http://c2.com/cgi/wiki?LispOnePointFive).

In particular, I found these quotes quite pertinent.

"I want to programmatically generate asts and feed those" "Even though this is what everybody does it's kinda nuts, why is the canonical code representation a STRING?!"

(not everyone does this, just algol derivatives)

"The ast is going to be designed along side the VM" "I need a tight feedback loop on the thing that I'm working on right now"

Wait, like every common lisp compiler ever? 30+ years behind the times yo.

"the code that you look at, that ought to be a reflection of the AST. The canonical thing ought to be the tree, the code is a view of it.... It's trees that are fundamental, that's what we work with"

Lol. Gotcha.

"something not offered by our tremendously entangled compiler, which doesn't even have a clean parse tree. It's comical. Try to get back to the source from what you get out of the scala parser. To me, the minimum test of a parser is that it parses!"

Lol. 'Comical' is definitely the right word.

"modifiability is paramount. If it isn't straightforward to modify, it will never be any good. It will never be fast. It will never be correct. And it will eventually be replaced by something modifiable... after consuming as many hours as you feed it."

Again, 30+ yrs behind the times[3]: http://article.gmane.org/gmane.comp.java.clojure.user/34272

So what does all this have to do with NixOS?

Much of the complexity in NixOS originates in the failure to address why it isn't a trivial modification on an existing project. Were they to have honestly sized up the problem they'd end up at something like: people suck at both reading & programming, also asts are fundamental. They'd then realize that at some point you have to choose some languages and leave others, because not everyone has something to offer and writing fully compliant parsers for all the languages involved is about as entertaining as building pyramids. When one has decided on a particular scheme, one could set about implementing "the ability to save and restore parts of unix's dependency graph".

But by failing to address this complexity NixOS inherits it, and in practice, adds to it by the 'invention' of /yet another/ language created for no particular reason. 800+ existing languages[4] and /none of them/ have the required properties? Nonsense. Allow me to anticipate your retorts:

"I need to know that my nix code is functional"

;; let's assume, for the sake of argument that you only want the symbols
;; `functional-lol', `design-antipattern', `the-type-gods-will-save-me' plus
;; strings to occur in your 'language' in addition to strings and keywords. You
;; know that the functions these symbols describe are functional and thus, a
;; valid program is functional.
;;
;; here is how you validate a 'nix' ast. 

(defun flatten (tree)
  "Traverses the tree in order, collecting non-null leaves into a list."
  (let (list)
    (labels ((traverse (subtree)
               (when subtree
                 (if (consp subtree)
                     (progn
                       (traverse (car subtree))
                       (traverse (cdr subtree)))
                     (push subtree list)))))
      (traverse tree))
    (nreverse list)))

(defvar myast '(functional-lol (design-antipattern "death") 
                               (the-type-gods-wont-save-me "destruction and chaos! also, bunnies")))

(defun validate (ast)
  (every (lambda (k) (or (keywordp k) (and (symbolp k)
             (member k '(functional-lol design-antipattern the-type-gods-will-save-me) :test 'eq)) 
            (stringp k)))
     (flatten ast)))

(validate myast) ; => nil

"I don't want to require a whole compiler!"

By failing to make use of existing programs NixOS not only adds a language (compiler/interpreter), but the requirement for an Emacs mode (+1 more for every other editor!), custom syntax highlighting, extra crap for auto-completion etc. etc. One should instead make use of what already exists. As it stands SLOCCOUNT reports 26.3k loc, divided across 7 languages being are used to introduce another for the nix project. This number of languages alone is enough to guarantee that no one will ever understand the whole codebase on a rolling basis and have enough time to do much else with their life.

A preceptable analogy to the natural languages springs forth: if we created them for the same reason so-called professional programmers create programming languages, each new subculture would require a whole new alphabet/glyph scheme, phonetic system, grammatical structures etc. Though some surface features may be shared across languages, the underlying semantics they represent would change in unpredictable and often incompatible ways.[5]

"So, you want us to write lisp? What if something better comes along?"

You are faced with a choice - either learn from those who came before you, or spend the rest of your life fighting monstrosities of your own creation, attempting to eke out some order in all the chaos, be it the SDF/BNF format of a language's syntax or otherwise. In any case, if something better comes along, transforming your s-expression code into $POWERFUL_NEW_NOTATION will be a straightforwards intern-level task[6].

Now to address some unanswered questions.

If you like the Nix concepts you could implement StumpWM the way you like it, or you could just walk away.

You don't say.

I'm also a little perplexed by the general tone and sense of entitlement but anyway... I'd like to address some of your points @gabriel-laddel .

You're in Mexico city and ask a bystander to direct you to the airport (your phone died, you lack a map and you're going to miss your plane unless you make it to the airport soon). He gives very specific instructions, which you write down and follow exactly. You reach the last instruction somewhere near the slums, missing your flight. As such, you swear loudly.

Is anyone entitled to correct directions? Of course not. I however, expect the /bare minimum/ of people giving correct directions and accurate descriptions of their projects, much like I expect my friends to brush their teeth regularly.

There's an SDF representation of the syntax in @edolstra's PHD thesis starting on page 64, though I'm not sure if that would be entirely up to date. It also can be seen as acting as a design document, and lays out the thought and central metaphors behind Nix/OS pretty clearly.

If I say "jump in front of the bus on my mark, ready? 1, 2, 3 - JUMP!" it is clear what I'm asking/ordering, what is less clear is why I'd be asking/ordering someone to jump in front of a bus.

:-) reminds me of classical Cathedral vs. Bazaar dilemma.

There are very few classics in the field of computing. We are at the very beginning of this journey and it hasn't even begun to get interesting yet. IMHO, the Cathedral vs. Bazaar isn't in anyway a dilemma or a classic. Linux was a failure 20 years ago, it is a failure today and any posturing otherwise is just that. See The UNIX-HATERS Handbook[0] for more information.

I find the contrast between these quotes accurately depicts the computing situation circa 2015.

The process of constructing instruction tables should be very fascinating. There need be no real danger of it ever becoming a drudge, for any processes that are quite mechanical may be turned over to the machine itself.

-- Turing, A. M., 1946, Proposed electronic calculator, report for National Physical Laboratory, Teddington

for the present-day darkness in the programmer's world the programmers themselves are responsible and nobody else.

-- Dijkstra

• Footnotes

[0] In actuality unix is mostly a pile of stupid, see: web.mit.edu/~simsong/www/ugh.pdf

[1]

Suppose you're trying to find the best way to structure your description of something. (Examples: choosing the structure for a computer program to perform some task; or choosing the structure for a theory of physics.)

What you hope to find is the natural structure of what you're describing — a structure that affords a really beautiful, simple description. When you strike the natural structure, a sort of resonance occurs, in which various subsidiary problems you may have had with your description just melt away, and the description practically writes itself.

But here's a problem that often occurs: You've got a structure that affords a pleasing approximate description. But as you try to tweak the description for greater precision, instead of the description getting simpler, as it should if you were really fine-tuning near the natural structure, instead the description gets more and more complicated. What has happened, I suggest, is that you've got a local optimum in solution space, instead of the global optimum of the natural structure: small changes in the structure won't work as well as the earlier approximation, and may not work at all, so fundamental improvement would require a large change to the structure.

-- http://fexpr.blogspot.com/2012/12/metaclassical-physics.html

[2] This scheme the surrounding ideas are so fundamental that they've sometime's referred to as "Maxwell's equations of Software". You can read more about Lisp and s-expressions elsewhere so I'll not repeat it here.

http://www.defmacro.org/ramblings/lisp.html http://www.michaelnielsen.org/ddi/lisp-as-the-maxwells-equations-of-software/ http://norvig.com/lispy.html http://norvig.com/lispy2.html

[3] This information re, Scala is interesting beyond the technicalities. Typesafe, the Scala company (has Martin Odersky as Chairman and Chief Scientist) has received 31 MM in funding. Coursera (who built everything in Scala) has received 85MM. I'll predict they'll eventually fail as they're out-competed by more intelligent adversaries, though both will (due the amount of funding and high-profile people involved, probably limp along for years to come).

Last I checked, Coursera already had fungus growing on it: something about, "usg-sponsored studies find that coursera is better than starving in Africa"

[4] http://en.wikipedia.org/wiki/List_of_programming_languages

[5]

The corresponding lisp narrative feels 'organic'. Consider the SBCL (steel bank common lisp) compiler. It is descended from CMUCL python compiler and uses some of its code, changing it as circumstances require (and of course, any transformations that needed automation were but a tree traversal away). Naggum also had some related thoughts:

There is a simple and elegant answer to this question: Just learn Common Lisp well first. New languages are exciting to people who know mostly new languages, so learn an old language before you learn new ones and get out of the maelstrom that will drown you in ever new languages that add nothing at all except some miniscule additional feature from another language that someone needed to make a whole new language to implement because he did not know (Common) Lisp to begin with. A "new" language that differs from the rest of the crop by one or a couple features is proof positive that both what it came from and what it has become are mutations about to die. There are tens if not hundreds of thousands of such "languages" that people have invented over the yeare, for all sorts of weird purposes where they just could not use whatever language they were already using, could not extend it, and could not fathom how to modify its tools without making a whole new language. They never stopped to think about how horribly wasteful this is, they just went on to create yet another language called Dodo, the Titanic, Edsel, Kyoto-agreement...

-- Erik Naggum, http://www.xach.com/naggum/articles/3206985430398054@naggum.net.html

[6] This could be largely automated, but that is a discussion for another day.

[wmertens]

Holy wall of text @gabriel-laddel!

I understand that you fully wanted to underline your points but I would have been happy with

• you didn't take existing source package management like CPAN into account 10 years ago
• you just scan for strings in the build output so you can not be 100% sure that the dependency graph is correct
• you invented Nix instead of using something like LISP

Those are your points right, or am I misunderstanding them/missing some?

Package management was indeed not built in from the beginning. The current approach is to convert instructions to npm and friends into nix package specifications. AFAIK this is in place for Haskell, Ruby, NodeJS, Python, Go and maybe Perl, not sure. This works okay but still has problems when semantics clash. The build flow is something like "generate nix expressions for dependencies, store in nixpkgs tree next to package and create nix expression for the package". Because of idempotency it doesn't matter if dependencies are mentioned many times, if they result in the same inputs hash they will be reused.

You can think of this as a very lazy static evaluation of needed packages out of hundreds of thousands of third-party packages. Considering the alternative of putting Nix expressions in other package management systems, I think this works pretty well.

Regarding the hash checking, the observation of no problems with it still stands. Dependency detection is quite robust and has low impact if incorrect.

• False positives (a dependency is falsely detected) can only be due to spurious unused references (it happens and we fix it if found), or 32+ byte ascii strings (hash+package name/version) occurring by chance in compiled code (extremely unlikely)
• False negatives (a dependency is not detected) can happen if the references are not stored in plain text, due to compression or encoding. I don't know of this happening in any of nixpkgs.
• The impact of incorrect dependencies is either increasing closure size, or missing dependencies when garbage collecting or installing a package from binary. The former is hardly problematic, the latter hasn't happened yet afaik, despite automatic GC running on many production systems.

So yes, both those things are only 99.9% solutions, but they are gaining 9s each year. The 100% solution seems infeasible to me in both cases.

For the last point, I think http://sandervanderburg.blogspot.com/2012/11/on-nix-and-gnu-guix.html articulates the issues nicely. Personally I prefer the advantages Nix gives over what Guix gets via Scheme.

Thank you for posting your thoughts, I still believe that NixOS and ecosystem are awesome and in a good position going forward.

On Sun Feb 08 2015 at 6:51:59 PM Gabriel Laddel notifications@github.com wrote:

Note1: I've read NixOS: A Purely Functional Linux Distribution and Nix: A Safe and Policy-Free System for Software Devlopment, but not the associated PHD thesis. The contents listing didn't indicate that it'd introduce me to any new ideas.

Note2: I've ignored forth and apl derivatives in this comment. Anyone familiar with will understand why upon reading it.

Note3: I posted and deleted this approximately ~5min ago - sorry, the using ``` as blockquote didn't work as expected.

Fundamentally, package management is composed of graph traversals and transformations. The goal that separates NixOS from other package managers, reproducible builds, can be rendered as "the ability to save and restore unix's dependency graph". This is a neat idea, but the implementation merely trades one set of problems for another. Many crucial design decisions that should have been throughly discussed were completely ignored. For example, nowhere in the design documents is it stated that every(?) programming language already has its own package manager(s) with its own idiosyncrasies that for the most part don't share concepts or code with the others. There are many good questions that originate from this fact, were one to acknowledge it. Examples: at what point is appropriate to leave package management up to a language's ecosystem? Is there ever a reason to? Can maintainers be convinced that the 'nixos way' is a good one and that packaging their code with a flag of some sort that produces a nix friendly build script is a reasonable request? Are we confident enough in nix to spend their time? Given that the NixOS plan is to modify build scripts and sources to its liking, what is the proper way to go about doing source-to-source transformations? Are there commercial offerings that solve this problem? The undiscussion of these issues is prototypical of the Nix design documents and documentation.

From NixOS: A Purely Functional Linux Distribution,

Runtime dependencies In Nix, we generally try to fix runtime dependencies at build time. This means that while a program may execute other programs or load dynamic libraries at runtime, the paths to those dependencies are hard-coded into the program at build time. For instance, for ELF executables, we set the RPATH in the executable such that it will find a statically determined set of library dependencies at runtime, rather than using a dynamic mechanism such as the LD_LIBRARY_PATH environment variable to look up libraries. This is important, because the use of such dynamic mechanisms makes it harder to run applications with conflicting dependencies at the same time (e.g., we might need Firefox linked against GTK 2.8 and Thunder bird linked against GTK 2.10). It also enhances determinism: a program will not suddenly behave differently on another system or under another user account because environment variables happen to be different. However, there is one case in NixOS and Nixpkgs of a library dependency that /must/ be overridable at runtime and cannot be fixed statically: the implementation of OpenGL to be used at runtime (libGL.so) which is hardware specific. We build applications that need OpenGL against Mesa, but add the impure (stateful) path /var/run/opengl-driver to the RPATH. The activate script syslinks that path to the actual OpenGL implementation selected by the configuration (e.g., nvidiaDriver) to allow programs to use it.

[...]

Build actions The Nix /model/ is that derivations are pure, that is, two builds of identical derivation should produce the same result in the Nix store. However, in contemporary operating systems, there is no way to actually enforce this model. Builders can use any impure source of information to produce the output, such as the system time, data downloaded from the network, or the current number of processes in the system as seen in /proc. It is trivial to construct a contrived builder that does such things. But build processes in the system as seen in /proc. But build processes generally do not, and instead are fairly determinisitc; impure influences such as the system time generally do not affect the runtime behavior of the package in question.

There are however frequent exceptions. First, many build processes are greatly affected by environment variables, such as PATH or CFLAGS. Therefore we clear the environment before starting a build (except for the attributes variable to a non-existent directory, because some derivations (such as Qt) try and read settings) from the user's home directory.

Second, almost all packages look for dependencies in impure locations such as /usr/bin and /usr/include. Indeed, the undeclared dependencies caused by this behavior are what motivated Nix in the first place: by storing packages in isolation from each other, we prevent undeclared build-time dependencies. In five years we haven't had a single instance of a package having an undeclared build-time dependency on another package /in the Nix store/, or having a runtime dependency in the Nix store not detected by the reference scanner. However, with Nix under other Linux distributions or operating systems, there have been numerous instances of packages affected by paths outside the Nix store. We prevent most of those impurities through a wrapper script around GCC and ld that ignores or fails on paths outside of the store. However, this cannot prevent undeclared dependencies such as direct calls to other programs, e.g., a Makefile running /usr/bin/yacc.

Since NixOS has no /bin, /usr and /lib, the effect of such impurities is greatly reduced. However, even in NixOS such impurities can occur. For instance, we recently encountered a problem with the build of the dbus package, which failed when /var/run/dbus didn't exist.

As a final example of impurity, some packages try to install files under a different location than $out. Nix causes such packages to /fail deterministically/ by executing builders under unprivileged UIDs that do not have write permisions to other store paths other than $out, let alone paths such as /bin. This packages must then be patched to make them well-behaved.

To ascertain how well these measures work in preventing impurities in NixOS, we performed two builds of the Nixpkgs collection on two different NixOS machines. This consisted of building 485 non-fetch url derivations. The output consisted of 165927 files and directories. Of these, there was only one /file name/ that differed between the two builds, namely in mono-1.1.4: a directory gac/IBM.Data.DB2/1.0.3008.37160_7c307b91aa13d208 versus 1.0.3008.40191_7c307b91aa13d280. The differing number is likely derived from system time.

We then compared each file. There we differences in 5059 files, or 3.4% of all regular files. We inspected the nature of the differences: almost all were caused by timestamps being encoded in files, such as in Unix object files archives or compiled Python code. 1048 Emacs Lisp files differed because the hostname of the build machines were stored in the output. Filtering out these and other files types that are known to contain timestamps as well (such as a build process inserting the build time in a C string). This hypothesis is strongly0 supported by the fact that of those, only 42 (or 0.03%) had different files sizes. None of these content differences have ever caused an observable difference in behavior.

How exactly does the reference scanner detect runtime dependencies? Regex and sed hackery? Or the correct way, parsing each language into its abstract syntax tree (ast), walking that and finding references to unix level dependencies (both static and dynamic)? I assume the former, as the latter is a great deal of work and would have been mentioned. Also, the only complete C99 parser (i.e., one that takes into account all the GCC extensions used in the linux kernel) I'm aware of is the haskell package Language.C.AST - and nix doesn't use haskell.

The problem with anything other than traversing a language's ast is that you're forever stuck with an approximation of the input language, resulting in false positives etc. This is not new information.

Much of the New Jersey approach is about getting away with less than is necessary to get the /complete/ job done. E.g., perl, is all about doing as little as possible that can approximate the full solution, sort of the entertainment industry's special effects and make-believe works, which for all practical purposes /is/ the real thing. Regular expressions is a pretty good approximation to actually parsing the implicit language of the input, too, but the rub with all these 90% solutions is that you have /no/ idea when they return the wrong value because the approximation destroys any ability to determine correctness. Most of the time, however, the error is large enough to cause a crash of some sort, but there is no way to do transactions, either, so a crash usually causes a debugging and rescue session to recover the state prior to the crash. This is deemed acceptable in the New Jersery approach. The reason they think this also /should/ be acceptable is that they believe that getting it exactly right is more expensive than fixing things after crashes. Therefore, the whole language must be optimized for getting the first approximations run fast.

-- http://www.xach.com/naggum/articles/3244575386963745@naggum.no.html

Notice that the paper says,

by storing packages in isolation from each other, we prevent undeclared build-time dependencies. In five years we haven't had a single instance of a package having an undeclared build-time dependency on another package /in the Nix store/, or having a runtime dependency in the Nix store not detected by the reference scanner.

but fails to mention how many people used NixOS over those 5 years, how many times undeclared build time dependencies /outside/ of the Nix store were encountered or how many packages were in the Nix store over this time period. Is this not relevant information? As Naggum stated above: "the rub with these 90% solutions is that you have /no/ idea when they return the wrong value becuase the approximation destroys any ability to determine correctness." If say, most of the NixOS users are NixOS developers (which they are) and the reference scanner doesn't walk the ast of the programs in question (which it doesn't), no reported failures means exactly nothing, because there will be many code paths not encountered by its users due to the sheer complexity of the software being interfaced with (e.g., how many NixOS +developers+ users use emacs docview? Not many apparently).

There are other issues with these papers I'm going to skip over, such as the clearing of environment variables (the proper thing to do imo is to gensym them to prevent unwanted interactions and study how they're being used - filter for 'functional' builds...), failure to discuss why a new project is needed in the first place (why can't you just perform this computation via an extension to an existing project?), total wtf issues with the papers (who are they written for? One who needs the concept of "file system" explained is going to have nfi what emacs lisp or ~/what/all/the/slashes/signify) etc.

The heart of the matter is that unix (ostensibly[0]) plays host to many different philosophies of how to go about programming. Languages differ in their approach to build systems, package management, syntax and foreign function interfaces. To change the way many software packages are built in a semi-automated, non-ast-walking fashion is insane. Not to discuss this in the design documents is insane. NixOS has 46 contributors and the documentation is sufficiently general as to give each contributor his own wildly different interpretation of the scope of the project. As far as I can tell, were one to extrapolate from the given information to a set of concrete requirements we see that NixOS plans to rewrite the build scripts for every version of every project on unix. Again, this is insane. The correct thing to do in this situation is to realize the utter impossibility of the task that has been set forth and re-evaluate one's approach.[1]

Let's examine the reference scanner. What is the correct way to approach the problem? We know regular expressions cannot respect the semantics of a language and thus are (unless used as a part of a larger parser) inappropriate for meta-programming (source-to-source transformations). Thus, we must work by manipulating the ast of various languages, and then pretty printing the altered code into the language's concrete syntax.

Most languages don't offer the ability to do this, and one examines into e.g., Clang, he will see all sorts of nonsense about concrete/surface syntax, context free grammars and compiler technologies. This certainly can't all be necessary (hint: it isn't).

Consider the following mathematical expression:

(3 + 2) * 8 / 3 * 3^6

Fully parenthesizing yields:

(((3 + 2) * 8) / (3 * 3^6))

When computers execute programs, or humans mathematics, the order of operations must be taken into account. one way to notate the previous expression to remove such ambiguities present in mathematical notation is to move operators to the front of each parenthesized list, passing the remaining elements as its arguments. This is known as fully-parenthesized prefix notation.

(/ (* (+ 3 2) 8) (* 3 (^ 3 6)))

This notational scheme has a direct mapping to the program's ast, which can be rendered as

http://i.imgur.com/8TO5VcK.png

See the pattern? An s-expression (fully parenthesized prefix notation) is merely a serialization for the ast. Thus, manipulations of the sexpr correspond to manipulations of the ast. Languages with the vocabulary to manipulate the s-expression structure can manipulate themselves (their ast) just as easily as any other thusly structured information. It follows from this that all software 'tooling' and translation tasks, transparently become trees e.g., find me all things that call this function, reference this variable, modify all build scripts to take into account the new &optional argument in the updated library, etc. are viewed quite transparently (and correctly) as tree traversals.[2]

For whatever reason, many find s-expressions distasteful and spend much of their lives attempting to add the same meta-programming facilities to ALGOL derived languages. They all have failed. Not because of any sort of mechanical failing of the computer, but the human's inability to fully comprehend the parsing and syntactical schemes they're able to create (Ruby's parser is 10k lines of C, Clang's is >100k loc. Note that the entirety of SBCL is ~300k loc, and has much fat that could be trimmed off - http://www.cliki.net/Lisp%20-%20Next%20Generation).

Scala is a notable failure in this regard. Watch this video:

http://www.youtube.com/watch?v=TS1lpKBMkgg

Pay attention to 37:39-42:50 and you'll get to see Paul Phillips flipping out over ir/asts (same thing!). He even states his plan for the next 25 years - attempt to solve a problem solved 50+ years ago ( http://c2.com/cgi/wiki?LispOnePointFive).

In particular, I found these quotes quite pertinent.

"I want to programmatically generate asts and feed those" "Even though this is what everybody does it's kinda nuts, why is the canonical code representation a STRING?!"

(not everyone does this, just algol derivatives)

"The ast is going to be designed along side the VM" "I need a tight feedback loop on the thing that I'm working on right now"

Wait, like every common lisp compiler ever? 30+ years behind the times yo.

"the code that you look at, that ought to be a reflection of the AST. The canonical thing ought to be the tree, the code is a view of it.... It's trees that are fundamental, that's what we work with"

Lol. Gotcha.

"something not offered by our tremendously entangled compiler, which doesn't even have a clean parse tree. It's comical. Try to get back to the source from what you get out of the scala parser. To me, the minimum test of a parser is that it parses!"

Lol. 'Comical' is definitely the right word.

"modifiability is paramount. If it isn't straightforward to modify, it will never be any good. It will never be fast. It will never be correct. And it will eventually be replaced by something modifiable... after consuming as many hours as you feed it."

Again, 30+ yrs behind the times[3]: http://article.gmane.org/gmane.comp.java.clojure.user/34272

So what does all this have to do with NixOS?

Much of the complexity in NixOS originates in the failure to address why it isn't a trivial modification on an existing project. Were they to have honestly sized up the problem they'd end up at something like: people suck at both reading & programming, also asts are fundamental. They'd then realize that at some point you have to choose some languages and leave others, because not everyone has something to offer and writing fully compliant parsers for all the languages involved is about as entertaining as building pyramids. When one has decided on a particular scheme, one could set about implementing "the ability to save and restore parts of unix's dependency graph".

But by failing to address this complexity NixOS inherits it, and in practice, adds to it by the 'invention' of /yet another/ language created for no particular reason. 800+ existing languages[4] and /none of them/ have the required properties? Nonsense. Allow me to anticipate your retorts:

"I need to know that my nix code is functional"

``common lisp

;; let's assume, for the sake of argument that you only want the symbols

;;functional-lol', design-antipattern',the-type-gods-will-save-me' plus

;; strings to occur in your 'language' in addition to strings and keywords. You ;; know that the functions these symbols describe are functional and thus, a ;; valid program is functional. ;; ;; here is how you validate a 'nix' ast.

(defun flatten (tree) "Traverses the tree in order, collecting non-null leaves into a list." (let (list) (labels ((traverse (subtree) (when subtree (if (consp subtree) (progn (traverse (car subtree)) (traverse (cdr subtree))) (push subtree list))))) (traverse tree)) (nreverse list)))

(defvar myast '(functional-lol (design-antipattern "death") (the-type-gods-wont-save-me "destruction and chaos! also, bunnies")))

(defun validate (ast) (every (lambda (k) (or (keywordp k) (and (symbolp k) (member k '(functional-lol design-antipattern the-type-gods-will-save-me) :test 'eq)) (stringp k))) (flatten ast)))

(validate myast) ; => nil

"I don't want to require a whole compiler!"

By failing to make use of existing programs NixOS not only adds a language (compiler/interpreter), but the requirement for an Emacs mode (+1 more for every other editor!), custom syntax highlighting, extra crap for auto-completion etc. etc. One should instead make use of what already exists. As it stands SLOCCOUNT reports 26.3k loc, divided across 7 languages being are used to introduce another for the nix project. This number of languages alone is enough to guarantee that no one will ever understand the whole codebase on a rolling basis and have enough time to do much else with their life.

A preceptable analogy to the natural languages springs forth: if we created them for the same reason so-called professional programmers create programming languages, each new subculture would require a whole new alphabet/glyph scheme, phonetic system, grammatical structures etc. Though some surface features may be shared across languages, the underlying semantics they represent would change in unpredictable and often incompatible ways.[5]

"So, you want us to write lisp? What if something better comes along?"

You are faced with a choice - either learn from those who came before you, or spend the rest of your life fighting monstrosities of your own creation, attempting to eke out some order in all the chaos, be it the SDF/BNF format of a language's syntax or otherwise. In any case, if something better comes along, transforming your s-expression code into will be a straightforwards intern-level task[6].

Now to address some unanswered questions.

If you like the Nix concepts you could implement StumpWM the way you like it, or you could just walk away.

You don't say.

I'm also a little perplexed by the general tone and sense of entitlement but anyway... I'd like to address some of your points @gabriel-laddel .

You're in Mexico city and ask a bystander to direct you to the airport (your phone died, you lack a map and you're going to miss your plane unless you make it to the airport soon). He gives very specific instructions, which you write down and follow exactly. You reach the last instruction somewhere near the slums, missing your flight. As such, you swear loudly.

Is anyone entitled to correct directions? Of course not. I however, expect the /bare minimum/ of people giving correct directions and accurate descriptions of their projects, much like I expect my friends to brush their teeth regularly.

There's an SDF representation of the syntax in @edolstra's PHD thesis starting on page 64, though I'm not sure if that would be entirely up to date. It also can be seen as acting as a design document, and lays out the thought and central metaphors behind Nix/OS pretty clearly.

If I say "jump in front of the bus on my mark, ready? 1, 2, 3 - JUMP!" it is clear what I'm asking/ordering, what is less clear is why I'd be asking/ordering someone to jump in front of a bus.

:-) reminds me of classical Cathedral vs. Bazaar dilemma.

There are very few classics in the field of computing. We are at the very beginning of this journey and it hasn't even begun to get interesting yet. IMHO, the Cathedral vs. Bazaar isn't in anyway a dilemma or a classic. Linux was a failure 20 years ago, it is a failure today and any posturing otherwise is just that. See The UNIX-HATERS Handbook[0] for more information.

I find the contrast between these quotes accurately depicts the computing situation circa 2015.

The process of constructing instruction tables should be very fascinating. There need be no real danger of it ever becoming a drudge, for any processes that are quite mechanical may be turned over to the machine itself.

-- Turing, A. M., 1946, Proposed electronic calculator, report for National Physical Laboratory, Teddington

for the present-day darkness in the programmer's world the programmers themselves are responsible and nobody else.

-- Dijkstra

• Footnotes

[0] In actuality unix is mostly a pile of stupid, see: web.mit.edu/~simsong/www/ugh.pdf

[1]

Suppose you're trying to find the best way to structure your description of something. (Examples: choosing the structure for a computer program to perform some task; or choosing the structure for a theory of physics.)

What you hope to find is the natural structure of what you're describing — a structure that affords a really beautiful, simple description. When you strike the natural structure, a sort of resonance occurs, in which various subsidiary problems you may have had with your description just melt away, and the description practically writes itself.

But here's a problem that often occurs: You've got a structure that affords a pleasing approximate description. But as you try to tweak the description for greater precision, instead of the description getting simpler, as it should if you were really fine-tuning near the natural structure, instead the description gets more and more complicated. What has happened, I suggest, is that you've got a local optimum in solution space, instead of the global optimum of the natural structure: small changes in the structure won't work as well as the earlier approximation, and may not work at all, so fundamental improvement would require a large change to the structure.

-- http://fexpr.blogspot.com/2012/12/metaclassical-physics.html

[2] This scheme the surrounding ideas are so fundamental that they've sometime's referred to as "Maxwell's equations of Software". You can read more about Lisp and s-expressions elsewhere so I'll not repeat it here. http://www.defmacro.org/ramblings/lisp.htmlhttp://www.michaelnielsen.org/ddi/lisp-as-the-maxwells-equations-of-software/http://norvig.com/lispy.htmlhttp://norvig.com/lispy2.html

[3] This information re, Scala is interesting beyond the technicalities. Typesafe, the Scala company (has Martin Odersky as Chairman and Chief Scientist) has received 31 MM in funding. Coursera (who built everything in Scala) has received 85MM. I'll predict they'll eventually fail as they're out-competed by more intelligent adversaries, though both will (due the amount of funding and high-profile people involved, probably limp along for years to come).

Last I checked, Coursera already had fungus growing on it: something about, "usg-sponsored studies find that coursera is better than starving in Africa"

[4] http://en.wikipedia.org/wiki/List_of_programming_languages

[5]

The corresponding lisp narrative feels 'organic'. Consider the SBCL (steel bank common lisp) compiler. It is descended from CMUCL python compiler and uses some of its code, changing it as circumstances require (and of course, any transformations that needed automation were but a tree traversal away). Naggum also had some related thoughts:

There is a simple and elegant answer to this question: Just learn Common Lisp well first. New languages are exciting to people who know mostly new languages, so learn an old language before you learn new ones and get out of the maelstrom that will drown you in ever new languages that add nothing at all except some miniscule additional feature from another language that someone needed to make a whole new language to implement because he did not know (Common) Lisp to begin with. A "new" language that differs from the rest of the crop by one or a couple features is proof positive that both what it came from and what it has become are mutations about to die. There are tens if not hundreds of thousands of such "languages" that people have invented over the yeare, for all sorts of weird purposes where they just could not use whatever language they were already using, could not extend it, and could not fathom how to modify its tools without making a whole new language. They never stopped to think about how horribly wasteful this is, they just went on to create yet another language called Dodo, the Titanic, Edsel, Kyoto-agreement...

-- Erik Naggum, http://www.xach.com/naggum/articles/3206985430398054@naggum.net.html

[6] This could be largely automated, but that is a discussion for another day.

—

Reply to this email directly or view it on GitHub https://github.com/NixOS/nixpkgs/issues/4952#issuecomment-73422170.

[gabriel-laddel]

I understand that you fully wanted to underline your points but I would have been happy with

• you didn't take existing source package management like CPAN into account 10 years ago
• you just scan for strings in the build output so you can not be 100% sure that the dependency graph is correct
• you invented Nix instead of using something like LISP

Uh... yeah, that's totally all the points I made.

Toodles.

[fare]

NB: I'm a Lisp guy who uses and likes NixOS, but isn't a big expert in NixOS.

Minor remarks: uiop:run-program has a now reliably portable :directory clause. If you only care about Unix, then you can use ; to separate commands in a single string. (If you want to support windows, & might work, but not all implementations can successfully use CMD.EXE on Windows, and /bin/sh isn't reliably present.)

NixOS isn't perfect, but is accepting patches.

If you're interested in static linking of libfixposix into sbcl images, I have recipes that need to be put together into actual code, that you may be interested in. If you want a NixOS package with dynamic linking the solution is to either enhance cffi so you can statically specify the search path, or have a wrapper that initializes the search environment around the binary.

Also, the solution to false negatives is that a package breaks if it's missing a dependency, so whoever is writing the package specification will notice and fix it — and thereafter it will build deterministically.

[gabriel-laddel]

Thanks for the info Francois. FTR, NixOS is fundamentally braindamaged. I found it was easier to create my own distribution than to hack around their crud. Details here:

http://gabriel-laddel.github.io/system.html http://gabriel-laddel.github.io/arsttep.html

On Fri, Jul 10, 2015 at 4:11 PM, François-René Rideau < notifications@github.com> wrote:

NB: I'm a Lisp guy who uses and likes NixOS, but isn't a big expert in NixOS.

Minor remarks: uiop:run-program has a now reliably portable :directory clause. If you only care about Unix, then you can use ; to separate commands in a single string. (If you want to support windows, & might work, but not all implementations can successfully use CMD.EXE on Windows, and /bin/sh isn't reliably present.)

NixOS isn't perfect, but is accepting patches.

If you're interested in static linking of libfixposix into sbcl images, I have recipes that need to be put together into actual code, that you may be interested in. If you want a NixOS package with dynamic linking the solution is to either enhance cffi so you can statically specify the search path, or have a wrapper that initializes the search environment around the binary.

Also, the solution to false negatives is that a package breaks if it's missing a dependency, so whoever is writing the package specification will notice and fix it — and thereafter it will build deterministically.

— Reply to this email directly or view it on GitHub https://github.com/NixOS/nixpkgs/issues/4952#issuecomment-120447628.

[jagajaga]

@gabriel-laddel :open_mouth: