load_strategies_json broken in Sage

[lducas]

If find myself unable to proerly load strategies when using fpyll through sage, and I get the trivial strategy with success proba 1. As a consequence, I suspect that sage currently does not provide BKZ2 but unpruned BKZ...

ipython
Python 2.7.16 (default, Oct  7 2019, 17:36:04) 
Type "copyright", "credits" or "license" for more information.

IPython 5.8.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from fpylll import *

In [2]: strats = load_strategies_json(BKZ.DEFAULT_STRATEGY)

In [3]: print strats[60]
Strategy< 60, (40), 0.29-0.50>

In [4]: print BKZ.DEFAULT_STRATEGY_PATH
/home/ducas/code/fpylll/fpylll-env/share/fplll/strategies

> ./sage 
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.9, Release Date: 2019-09-29                     │
│ Using Python 2.7.15. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
sage: from fpylll import *
sage: strats = load_strategies_json("/home/ducas/code/fpylll/fpylll-env/share/fplll/strategies/default.json")
sage: print strats[60]
Strategy< 60, (), 1.00-1.00>

[lducas]

Here is a MonkeyPatch for hotfix in sage:

import json
from fpylll.fplll.bkz_param import Strategy
from fpylll import BKZ
import os

path_strat = os.getenv("SAGE_ROOT")+"/local/share/fplll/strategies/default.json"

with open(path_strat) as json_data:
     data = json.load(json_data)

strategies = 91 * [None]
for datum in data:
    b = datum["block_size"]
    prun = datum["pruning_parameters"]
    prep = datum["preprocessing_block_sizes"]
    strat = Strategy(b, prep, prun)
    strategies[b] = strat

BKZ.DEFAULT_STRATEGY = strategies

[malb]

I just built Sage from source:

Forcing sage-location, probably because a new package was installed.
Cleaning up, do not interrupt this.
Done cleaning.
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.9, Release Date: 2019-09-29                     │
│ Using Python 2.7.15. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
sage: from fpylll import *
sage: strats = load_strategies_json(BKZ.DEFAULT_STRATEGY)
sage: print strats[60]
Strategy< 60, (40), 0.29-0.50>

Did you move your installation after compilation?

[lducas]

I did not compiled sage. But more to the point, this issue here is not about finding the file, but about reading it properly: my sage example is set to fetch the same file than standalone fpylll, and stills privinde a different output.

[malb]

Mhh, I can't reproduce it:

./sage 
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.9, Release Date: 2019-09-29                     │
│ Using Python 2.7.15. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
sage: from fpylll import *
sage: print load_strategies_json("/home/malb/projects/lattices/fplll/strategies/default.json")[60]
Strategy< 60, (40), 0.29-0.50>

[lducas]

Well, I do have this same issue on 3 different machines (all linuxes: Fedora and ubuntu). Though the common denominator is that I use sage binaries rather than compiling it myself. Here on another machine with sage 8.8:

> sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.8, Release Date: 2019-06-26                     │
│ Using Python 2.7.15. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
sage: from fpylll import *
sage: print load_strategies_json("/home/ducas/code/fplll/strategies/default.json")[60]
Strategy< 60, (), 1.00-1.00>
sage: 

[malb]

With a precompiled binary, I can reproduce it:

┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.9, Release Date: 2019-09-29                     │
│ Using Python 2.7.15. Type "help()" for help.                       │
└────────────────────────────────────────────────────────────────────┘
sage: from fpylll import *
sage: BKZ.DEFAULT_STRATEGY_PATH
'/bulk/home/malb/software/SageMath/local/share/fplll/strategieparse error - unpreprocessing_blpruning_parameteError: gptr == nulls'
sage: BKZ.DEFAULT_STRATEGY
'/bulk/home/malb/software/SageMath/local/share/fplll/strategieparse error - unpreprocessing_blpruning_parameteError: gptr == nulls/default.json'
sage: print load_strategies_json("/home/malb/projects/lattices/fplll/strategies/default.json")[60]
Strategy< 60, (), 1.00-1.00>

[malb]

Looks like a Sage bug to me, but I'm at a loss.

[lducas]

Sure, its a sage bug. But we should be concerned about it. I think most of the community uses BKZ through sage, and currently they are using BKZ when they think they are using BKZ2. It can imply incorrect conclusion in research papers.

[malb]

I agree we should worry about it, I've opened https://trac.sagemath.org/ticket/28664 and will send an e-mail to [sage-devel]