17 internet doctest failing in findstat.py

[seblabbe]

findstat.org has now a proper api. In this ticket we switch to this api, and create a hierarchy of classes that better models what findstat provides.

In particular, this allows search for maps, iterating over all statistics or maps with given domain, easier access to search for distributions, etc.

Restricted to the capabilities of the old interface, the usage remains the same.

We also fix all the failing doctest.

Original ticket description

Using SageMath version 9.2.beta2, Release Date: 2020-06-26, the command

sage -t --optional=sage,internet src/sage/databases/findstat.py

gives

**********************************************************************
7 items had failures:
   4 of  16 in sage.databases.findstat
   3 of  10 in sage.databases.findstat.FindStat
   1 of  11 in sage.databases.findstat.FindStat.__call__
   1 of  10 in sage.databases.findstat.FindStatCollection.in_range
   3 of   5 in sage.databases.findstat.FindStatStatistic.first_terms
   3 of   5 in sage.databases.findstat.FindStatStatistic.generating_functions
   2 of   4 in sage.databases.findstat.FindStatStatistic.oeis_search
    [249 tests, 17 failures, 41.24 s]
----------------------------------------------------------------------
sage -t src/sage/databases/findstat.py  # 17 doctests failed
----------------------------------------------------------------------

CC: @mantepse @stumpc5

Component: combinatorics

Keywords: FindStat

Author: Martin Rubey

Branch/Commit: 52c576e

Reviewer: Sébastien Labbé

Issue created by migration from https://trac.sagemath.org/ticket/28864

[fchapoton]

comment:1

any idea of what the problem could be ? some problem for empty shape, maybe ?

[mantepse]

comment:2

Essentially the problem is (very very likely) that there is a new collection on findstat.org (skew partitions), which is not reflected in the sage interface (yet).

Very unfortunately, findstat.org itself has (technical) problems which I was unable to resolve so far.

On top of that, investing time into the old code is going to delay the deployment of the new and shiny and proper api for the website even further, although it is almost done (since an embarassingly long time).

One way to speed up the process would be to decide to drop support for the old way to access findstat completely. This would mean that, that I would switch findstat.org and the sage interface to the new api simultaneously, and that old versions of sage could not use findstat anymore. I doubt that this is a very good idea, but it is a possibility.

[mantepse]

comment:3

a brief update:

• I am now finished with implementing a proper api on the server side of FindStat, it is currently in testing - findstat.org itself is using this api since 31.12.2019.

• My next task is to rewrite the sage-findstat interface, I will use this as a testcase to check whether the api is well designed. I intend to publish the api and the new interface at the same time.

In the meantime, the sage interface should be fully functional.

[embray]

comment:4

Ticket retargeted after milestone closed

[mantepse]

Branch: u/mantepse/1_internet_doctest_failing_in_findstat_py

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

eb29c9e

begin fixing doctests, add some more infrastructure

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Commit: eb29c9e

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

d1b2f29

make all statistics by domain accessible via findstat

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from eb29c9e to d1b2f29

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from d1b2f29 to 696e79d

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

696e79d

make cache for statistics global

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

5070c5f

fix iterating over maps and statistics, fix doctests

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 696e79d to 5070c5f

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

64b685c

fix and improve argument parsing in findmap

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 5070c5f to 64b685c

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 64b685c to b5ad6e9

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

b5ad6e9

fix 3rd positional argument for findmap, fix doctests for offline

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from b5ad6e9 to 243d0b8

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

243d0b8

make tab completion work

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 243d0b8 to 725e6a3

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

725e6a3

doc fixes

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

e7c68c4

py3 fixes

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 725e6a3 to e7c68c4

[stumpc5]

comment:15

Here are some first comments:

• """ There is a new collection available at The Combinatorial Statistic Finder (https://www.findstat.org/): Signed permutations. To use it with this interface, it has to be added to the dictionary SupportedFindStatCollections in src/sage/databases/findstat.py of the SageMath distribution. Please open a ticket on trac! """

  Would it be possible to automatize this? Otherwise, every user will see this on startup and likely get confused.

• I would prefer a single entry point (maybe FindStat with FindStat.find_statistic and FindStat.find_map?). If I know findstat but not findmap, I might completely miss the latter? Also, using FindStat? would immediately give instructions how to proceed.

• (This is not for this ticket but for the backend:) Queries from the interface are not yet logged in the findstat log.

• 41: St001489oMp00081oMp00059 (quality [100, 100]) The term quality is not obviously explained. In the doc, it might be referred to as "a number which says how many values...". a proper definition of this quality might be helpful.

• The term distribution is not explained how to be used in the input section of findstat: it is twice referred to as "distribution must be None".

• It remains unclear (I think, to a new user) what "a new statistic" means. I would suggest to not show this as one of the "results" of a DababaseQuery. Instead, I would suggest to have this available as a method submit_as_new_statistic.

  • This method fails if the input datum is not a statistic (but a distribution).
  • If there is no result, the user gets prompted a message that this method is available in case she wants to submit the data.

• The term offset is not exlained, is it?

• What about two methods matching values and unmatched_values ?

  • Each of these could consist of lists of pairs of the form ( ( elt1, elt2, elt3 ), val ).

• Finally: Since the output might be long and confusing, I think it would be great if FindStatMatchingStatistic had a method info or detailed_info or alike which would in detail present this match:

  sage: A = findstat("Permutations", distribution = lambda pi: pi.number_of_descents())
  sage: a = A[37]; a
  St001169oMp00012oMp00061 (quality [100, 100])
  sage: a.info()
  After applying
  
  Mp00061: to increasing tree
  Mp00012: to Dyck path: up step, left tree, down step, right tree
  (see `a.maps()` for details)
  
  your input data matches
  
  St001169: Number of simple modules with projective dimension at least two in the corresponding Nakayama algebra.
  (see `a.statistic()` for details)
  
  with offset 0 in xxx / yyy many values and xxx/yyy many different values (see `a.matching_values()` and `a.unmatched_values()` for details).

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from e7c68c4 to 503bd29

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

503bd29

documentation improvements, new method info for matches

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

8e3f237

minor doc fixes

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 503bd29 to 8e3f237

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

0f1642b

restructure to provide generating functions also for compound statistics

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 8e3f237 to 0f1642b

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 0f1642b to 817cb84

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

817cb84

better separation of concerns, add docstrings

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 817cb84 to 80736f2

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

80736f2

fix a few bugs

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

bfffe26

restrict doctests to fewer statistics, fix handling of None as value, improve call

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 80736f2 to bfffe26

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

9a144f2

submit only FINDSTAT_MAX_SUBMISSION_VALUES values, refactor function to get source

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from bfffe26 to 9a144f2

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

e786fbe

provide more doctests

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 9a144f2 to e786fbe

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from e786fbe to 4a84e3f

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

e1b20fa	Merge branch 'develop' into t/28864/1_internet_doctest_failing_in_findstat_py
ccfddb6	Merge branch 'u/mantepse/1_internet_doctest_failing_in_findstat_py' of git://trac.sagemath.org/sage into t/28864/1_internet_doctest_failing_in_findstat_py
4a84e3f	use requests.get instead of urlopen, refactor FindStatStatisticQuery._generating_functions_dict

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 4a84e3f to da59574

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

da59574

try to fix unicode/py2/py3 issues

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

3a0f0d6

provide remaining doctests

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from da59574 to 3a0f0d6

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Branch pushed to git repo; I updated commit sha1. New commits:

5d2cce4

fix isinstance for Semistandard tableaux, add signed permutations, fix doctests

[7ed8c4ca-6d56-4ae9-953a-41e42b4ed313]

Changed commit from 3a0f0d6 to 5d2cce4

[mantepse]

Author: Martin Rubey

[mantepse]

Changed keywords from none to FindStat