Jsoo sherlodoc - Githubissues

EmileTrotignon commented 1 year ago

This contains a lot of changes. I will try to broadly describe them.

Compatibility with odoc, that is this version : https://github.com/ocaml/odoc/pull/972
Searching things that are not values : modules, types, constructors, and type-based search where applicable
js_of_ocaml version that conforms to the protocol outlined by odoc pr 972.
There is a new data structure to store the results that has better performance both in space and time.
There is a cli version of the search that was written for testing purposes but that can probably be used as a standalone tool.

I have probably forgotten some things, I will come back and add details later.

EmileTrotignon commented 1 year ago

I have added a bunch of comments, which on new files is better than updating the PR description.

EmileTrotignon commented 11 months ago

I have tested building search databases on a lot of packages, and I have no bug to report. Here is the list of packages : The database were built separately, using the odocls given by odig. The biggest database was for octez-proto-libs at 111M, but most of the sizes are reasonable, rarely exceeding 10M, and being around 1M most of the time. This is before compression, so way less data is going to be actually transmitted through the network.

absolute.0.3
accessor.v0.16.0
aches.1.0.0
aches-lwt.1.0.0
acp4.1.0.1
alcotest.1.7.0
alsa.0.3.0
alt-ergo.2.5.2
alt-ergo-lib.2.5.2
alt-ergo-parsers.2.5.2
ancient.0.9.1
angstrom.0.15.0
apron.v0.9.14
apronext.1.0.4
arrakis.1.0.0
art.0.2.0
asetmap.0.8.1
asn1-combinators.0.2.6
astring.0.8.5
async.v0.16.0
async_kernel.v0.16.0
async_rpc_kernel.v0.16.0
async_unix.v0.16.0
async_websocket.v0.16.0
b0.0.0.5
base.v0.16.3
base-bigarray.base
base-bytes.base
base-threads.base
base-unix.base
base64.3.5.1
base_bigstring.v0.16.0
base_quickcheck.v0.16.0
batteries.3.7.1
bheap.2.0.0
bigarray-compat.1.1.0
bigarray-overlap.0.2.1
bigstring.0.3
bigstringaf.0.9.1
bin_prot.v0.16.0
bls12-381.18.0
bos.0.2.1
brr.0.0.6
bst.7.0.1
ca-certs.0.2.3
camlidl.1.11
camlp-streams.5.0.1
camlzip.1.11
caqti.1.9.0
caqti-lwt.1.9.0
checkseum.0.5.2
chrome-trace.3.11.1
class_group_vdf.0.0.4
cmdliner.1.2.0
cohttp.5.3.0
cohttp-lwt.5.3.0
cohttp-lwt-unix.5.3.0
conduit.6.2.0
conduit-lwt.6.2.0
conduit-lwt-unix.6.2.0
conf-alsa.1
conf-autoconf.0.1
conf-cmake.1
conf-g++.1.0
conf-gmp.4
conf-gmp-powm-sec.3
conf-hidapi.0
conf-libev.4-12
conf-libffi.2.0.0
conf-libssl.4
conf-mpfr.3
conf-perl.2
conf-pkg-config.3
conf-rust.0.1
conf-rust-2021.1
conf-sdl2.1
conf-which.1
conf-zlib.1
core.v0.16.2
core_kernel.v0.16.0
core_unix.v0.16.0
cpm.12.2.0
cppo.1.6.9
cpu.2.0.0
cryptokit.1.16.1
csexp.1.5.2
cstruct.6.2.0
cstruct-lwt.6.2.0
ctypes.0.20.2
ctypes-foreign.0.18.0
ctypes_stubs_js.0.1
data-encoding.0.7.1
decompress.1.5.3
digestif.1.1.4
dolmen.0.9
dolmen_loop.0.9
dolmen_type.0.9
dolog.6.0.0
domain-name.0.4.0
dream.1.0.0~alpha5
dream-httpaf.1.0.0~alpha2
dream-pure.1.0.0~alpha2
dune.3.10.0
dune-build-info.3.11.1
dune-configurator.3.11.1
dune-private-libs.3.11.1
dune-rpc.3.11.1
dune-site.3.11.1
duration.0.2.1
dyn.3.11.1
either.1.0.0
eqaf.0.9
expect_test_helpers_core.v0.16.0
ezjsonm.1.3.0
faraday.0.8.2
faraday-lwt.0.8.2
faraday-lwt-unix.0.8.2
fiber.3.7.0
fieldslib.v0.16.0
fix.20230505
fmlib.0.5.6
fmlib_browser.0.5.6
fmlib_js.0.5.6
fmlib_parse.0.5.6
fmlib_pretty.0.5.6
fmlib_std.0.5.6
fmt.0.9.0
fpath.0.7.3
gen.1.1
gg.1.0.0
gmap.0.3.0
graphql.0.14.0
graphql-lwt.0.14.0
graphql_parser.0.14.0
hacl-star.0.7.1
hacl-star-raw.0.7.1
hashcons.1.3
hex.1.5.0
hidapi.1.1.2
higher_kinded.v0.16.0
hkdf.1.0.4
hmap.0.8.1
htmlit.0.1.0
index.1.6.1
int_repr.v0.16.0
integers.0.7.0
integers_stubs_js.1.0
ipaddr.5.5.0
ipaddr-sexp.5.5.0
irmin.3.7.2
irmin-pack.3.7.2
jane-street-headers.v0.16.0
js_of_ocaml.5.4.0
js_of_ocaml-compiler.5.4.0
js_of_ocaml-ppx.5.4.0
js_of_ocaml-toplevel.5.4.0
json-data-encoding.0.12.1
json-data-encoding-bson.0.12.1
jsonm.1.0.2
jst-config.v0.16.0
ke.0.6
ledgerwallet.0.3.0
ledgerwallet-tezos.0.3.0
libabsolute.0.1
line_oriented.1.3.0
logs.0.7.0
lru.0.3.1
lwt.5.7.0
lwt-canceler.0.3
lwt-exit.1.0
lwt-watcher.0.2
lwt_ppx.2.1.0
lwt_ssl.1.2.0
macaddr.5.5.0
magic-mime.1.3.1
matplotlib.0.2
mdx.2.3.1
menhir.20230608
menhirLib.20230608
menhirSdk.20230608
merlin-lib.4.12-414
minicli.5.0.2
mirage-clock.4.2.0
mirage-crypto.0.11.2
mirage-crypto-ec.0.11.2
mirage-crypto-pk.0.11.2
mirage-crypto-rng.0.11.2
mirage-crypto-rng-lwt.0.11.2
mlgmpidl.1.2.15
mtime.1.4.0
multipart_form.0.5.0
multipart_form-lwt.0.5.0
num.1.4
ocaml.4.14.1
ocaml-base-compiler.4.14.1
ocaml-compiler-libs.v0.12.4
ocaml-config.2
ocaml-migrate-parsetree.2.4.0
ocaml-options-vanilla.1
ocaml-syntax-shims.1.0.0
ocaml-version.3.6.2
ocaml_intrinsics.v0.16.0
ocamlbuild.0.14.2
ocamlc-loc.3.11.1
ocamlfind.1.9.6
ocamlformat.0.26.1
ocamlformat-lib.0.26.1
ocamlformat-rpc-lib.0.26.1
ocamlgraph.2.1.0
ocp-indent.1.8.1
ocp-ocamlres.0.4
ocplib-endian.1.2
ocplib-simplex.0.5
octez.18.0
octez-accuser-Proxford.18.0
octez-accuser-PtNairob.18.0
octez-alcotezt.18.0
octez-baker-Proxford.18.0
octez-baker-PtNairob.18.0
octez-client.18.0
octez-codec.18.0
octez-crawler.18.0
octez-dac-client.18.0
octez-dac-node.18.0
octez-distributed-internal.18.0
octez-distributed-lwt-internal.18.0
octez-injector.18.0
octez-l2-libs.18.0
octez-libs.18.0
octez-node.18.0
octez-node-config.18.0
octez-proto-libs.18.0
octez-protocol-000-Ps9mPmXa-libs.18.0
octez-protocol-001-PtCJ7pwo-libs.18.0
octez-protocol-002-PsYLVpVv-libs.18.0
octez-protocol-003-PsddFKi3-libs.18.0
octez-protocol-004-Pt24m4xi-libs.18.0
octez-protocol-005-PsBabyM1-libs.18.0
octez-protocol-006-PsCARTHA-libs.18.0
octez-protocol-007-PsDELPH1-libs.18.0
octez-protocol-008-PtEdo2Zk-libs.18.0
octez-protocol-009-PsFLoren-libs.18.0
octez-protocol-010-PtGRANAD-libs.18.0
octez-protocol-011-PtHangz2-libs.18.0
octez-protocol-012-Psithaca-libs.18.0
octez-protocol-013-PtJakart-libs.18.0
octez-protocol-014-PtKathma-libs.18.0
octez-protocol-015-PtLimaPt-libs.18.0
octez-protocol-016-PtMumbai-libs.18.0
octez-protocol-017-PtNairob-libs.18.0
octez-protocol-018-Proxford-libs.18.0
octez-protocol-alpha-libs.18.0
octez-protocol-compiler.18.0
octez-proxy-server.18.0
octez-shell-libs.18.0
octez-signer.18.0
octez-smart-rollup-client-Proxford.18.0
octez-smart-rollup-client-PtNairob.18.0
octez-smart-rollup-node-lib.18.0
octez-smart-rollup-node-Proxford.18.0
octez-smart-rollup-node-PtNairob.18.0
octez-smart-rollup-wasm-debugger.18.0
octez-version.18.0
odig.0.0.9
odoc.2.3.0
odoc-parser.2.3.0
opam-core.2.1.5
optint.0.3.0
ordering.3.11.1
parany.14.0.1
parsexp.v0.16.0
pbkdf.1.2.0
pecu.0.6
picasso.0.4.0
pp.1.2.0
pp_loc.2.1.0
pprint.20230830
ppx_assert.v0.16.0
ppx_base.v0.16.0
ppx_bench.v0.16.0
ppx_bin_prot.v0.16.0
ppx_blob.0.7.2
ppx_cold.v0.16.0
ppx_compare.v0.16.0
ppx_custom_printf.v0.16.0
ppx_derivers.1.2.1
ppx_deriving.5.2.1
ppx_disable_unused_warnings.v0.16.0
ppx_enumerate.v0.16.0
ppx_expect.v0.16.0
ppx_fields_conv.v0.16.0
ppx_fixed_literal.v0.16.0
ppx_globalize.v0.16.0
ppx_hash.v0.16.0
ppx_here.v0.16.0
ppx_ignore_instrumentation.v0.16.0
ppx_import.1.10.0
ppx_inline_test.v0.16.0
ppx_irmin.3.7.2
ppx_jane.v0.16.0
ppx_let.v0.16.0
ppx_log.v0.16.0
ppx_module_timer.v0.16.0
ppx_optcomp.v0.16.0
ppx_optional.v0.16.0
ppx_pipebang.v0.16.0
ppx_repr.0.7.0
ppx_sexp_conv.v0.16.0
ppx_sexp_message.v0.16.0
ppx_sexp_value.v0.16.0
ppx_stable.v0.16.0
ppx_stable_witness.v0.16.0
ppx_string.v0.16.0
ppx_tydi.v0.16.0
ppx_typerep_conv.v0.16.0
ppx_variants_conv.v0.16.0
ppx_yojson_conv_lib.v0.16.0
ppxlib.0.31.0
prbnmcn-basic-structures.0.0.1
prbnmcn-linalg.0.0.1
prbnmcn-stats.0.0.6
prettym.0.0.3
pringo.1.3
progress.0.2.1
prometheus.1.2
prometheus-app.1.2
protocol_version_header.v0.16.0
psmt2-frontend.0.4.0
psq.0.2.1
ptime.1.1.0
pure-splitmix.0.3
pyml.20220905
qcheck-alcotest.0.21.2
qcheck-core.0.21.2
re.1.11.0
redis.0.7.1
repr.0.7.0
resto.1.2
resto-acl.1.2
resto-cohttp.1.2
resto-cohttp-client.1.2
resto-cohttp-self-serving-client.1.2
resto-cohttp-server.1.2
resto-directory.1.2
result.1.5
ringo.1.0.0
rresult.0.7.0
rusage.1.0.0
secp256k1-internal.0.4.0
sedlex.3.2
semaphore-compat.1.0.1
seq.base
seqes.0.2
sexp_pretty.v0.16.0
sexplib.v0.16.0
sexplib0.v0.16.0
spawn.v0.15.1
spelll.0.4
splittable_random.v0.16.0
ssl.0.7.0
stdcompat.19
stdint.0.7.2
stdio.v0.16.0
stdlib-shims.0.3.0
stdune.3.11.1
stringext.1.6.0
tar.2.6.0
tar-unix.2.6.0
terminal.0.2.1
textutils.v0.16.0
textutils_kernel.v0.16.0
tezos-benchmark.18.0
tezos-dac-client-lib.18.0
tezos-dac-lib.18.0
tezos-dac-node-lib.18.0
tezos-dal-node-lib.18.0
tezos-dal-node-services.18.0
tezos-lwt-result-stdlib.17.3
tezos-protocol-000-Ps9mPmXa.18.0
tezos-protocol-001-PtCJ7pwo.18.0
tezos-protocol-002-PsYLVpVv.18.0
tezos-protocol-003-PsddFKi3.18.0
tezos-protocol-004-Pt24m4xi.18.0
tezos-protocol-005-PsBABY5H.18.0
tezos-protocol-005-PsBabyM1.18.0
tezos-protocol-006-PsCARTHA.18.0
tezos-protocol-007-PsDELPH1.18.0
tezos-protocol-008-PtEdo2Zk.18.0
tezos-protocol-008-PtEdoTez.18.0
tezos-protocol-009-PsFLoren.18.0
tezos-protocol-010-PtGRANAD.18.0
tezos-protocol-011-PtHangz2.18.0
tezos-protocol-012-Psithaca.18.0
tezos-protocol-013-PtJakart.18.0
tezos-protocol-014-PtKathma.18.0
tezos-protocol-015-PtLimaPt.18.0
tezos-protocol-016-PtMumbai.18.0
tezos-protocol-017-PtNairob.18.0
tezos-protocol-018-Proxford.18.0
tezos-protocol-alpha.18.0
tezos-proxy-server-config.18.0
tezos-rust-libs.1.6
tezos-sapling-parameters.1.1.0
tezt.3.1.1
tezt-tezos.18.0
tgls.0.8.6
time_now.v0.16.0
timezone.v0.16.0
tls.0.17.1
tls-lwt.0.17.1
topkg.1.0.7
tsdl.1.0.0
typerep.v0.16.0
tyxml.4.6.0
unstrctrd.0.3
uri.4.4.0
uri-sexp.4.4.0
uucd.15.1.0
uucp.15.1.0
uuidm.0.9.8
uunf.15.1.0
uuseg.15.1.0
uutf.1.0.3
variantslib.v0.16.0
vector.1.0.0
vector3.1.0.0
vg.0.9.4
webbrowser.0.6.1
x509.0.16.5
xdg.3.11.1
xmlm.1.4.0
yaml.3.1.0
yojson.2.1.1
zarith.1.12
zarith_stubs_js.v0.16.0

art-w commented 9 months ago

Thanks again for all the work! :heart:

It's been deployed to doc.sherlocode.com a few days ago and seems to work well enough... so let's merge! Any remaining issues can be addressed in follow-up PRs :)

art-w / sherlodoc

Jsoo sherlodoc #4