search

iu-parfunc / sc-haskell

A survey on the topic of sequentially consistent Haskell.
3 stars 0 forks source link

Survey to measure Safe Haskell Stackage "response curve" #10

Open rrnewton opened 8 years ago

rrnewton commented 8 years ago

Here's the methodology. It's described at the package granularity, but it may be worth dropping down to the module granularity and (mostly) ignoring packages.

  1. Create a list of all N stackage packages, sorted by "number of transitive reverse dependencies".
  2. Compile stackage with -fpackage-trust and figure out what whitelist of packages is needed for the baseline level of safety already present.
  3. Establish a baseline labeling of each package as safe/trustworthy/unsafe. It may help to use the three warning flags -Wunsafe, -Wsafe, -Wtrustworthy-safe.
  4. for i = 1 .. N, increase the TCB by marking package i as Trustworthy. Attempt to compile all packages downstream from i with safety required (unless they are already trusted), and see how many "flip". Record the total count of safe/trustworthy/unsafe packages.
  5. Plot how the count of safe/trustworthy/unsafe packages changes from 1 to N.
rrnewton commented 8 years ago

Actually for 1..N isn't quite accurate. If marking an earlier package as TrustWorthy causes a downstream dependency to become safe, then that downstream pkg can be removed from the "worklist" of packages to "trustify" in the experiment.

rrnewton commented 8 years ago

P.S. The above analysis can also be run filtering stackage to only the set of pkgs with benchmarks, plus their dependencies. That analysis is most relevant to our benchmarking problem, but the broader analysis is most relevant to the broader question of "can Safe Haskell be made real".