mikemccand / luceneutil

Various utility scripts for running Lucene performance tests
Apache License 2.0
205 stars 115 forks source link

Make auto generated index name shorter #198

Open zhaih opened 2 years ago

zhaih commented 2 years ago

See this comment, the auto generated index name could get overlong and JFR is complaining about it. Altho it can be easily overcame by just give the index a dedicated name, but anyway I think make the auto generated index name shorter is still necessary: right now we're trying to list the index spec as part of the name but that will make it extremely hard to open any file using this index name. Perhaps we can just use timestamp as generated index name, and then put the specs in some separated file inside index directory?

slow-J commented 1 year ago

Big +1 to this.

Right now, luceneutil does not work "out of the box" without any local code changes due to the File name too long error that jfr throws.

To reproduce:

  1. Set up luceneutil as per the readme.
  2. Run python3 src/python/localrun.py -source wikimediumall -r
  3. Error gets thrown:
    Create indices:
    /local/home/jslowins/upstream_bench/lucene_bench_home/indices/wikimediumall.lucene_candidate.candidate.facets.taxonomy:Date.taxonomy:Month.taxonomy:DayOfYear.sortedset:Date.sortedset:Month.sortedset:DayOfYear.taxonomy:RandomLabel.sortedset
    :RandomLabel.Lucene90.Lucene99.dvfields.nd33.3326M: now create
    cd /local/home/jslowins/upstream_bench/lucene_bench_home/lucene_candidate/lucene/benchmark
    log /local/home/jslowins/upstream_bench/lucene_bench_home/logs/baseline_vs_patch.wikimediumall.lucene_candidate.candidate.facets.taxonomy:Date.taxonomy:Month.taxonomy:DayOfYear.sortedset:Date.sortedset:Month.sortedset:DayOfYear.taxonomy:
    RandomLabel.sortedset:RandomLabel.Lucene90.Lucene99.dvfields.nd33.3326M.log
    WARNING: Using incubator modules: jdk.incubator.vector
    [0.302s][error][jfr,startup] Could not start recording, not able to write to file /local/home/jslowins/upstream_bench/lucene_bench_home/logs/bench-index-baseline_vs_patch-wikimediumall.lucene_candidate.candidate.facets.taxonomy:Date.taxonomy
    :Month.taxonomy:DayOfYear.sortedset:Date.sortedset:Month.sortedset:DayOfYear.taxonomy:RandomLabel.sortedset:RandomLabel.Lucene90.Lucene99.dvfields.nd33.3326M.jfr. /local/home/jslowins/upstream_bench/lucene_bench_home/logs/bench-index-baselin
    e_vs_patch-wikimediumall.lucene_candidate.candidate.facets.taxonomy:Date.taxonomy:Month.taxonomy:DayOfYear.sortedset:Date.sortedset:Month.sortedset:DayOfYear.taxonomy:RandomLabel.sortedset:RandomLabel.Lucene90.Lucene99.dvfields.nd33.3326M.jf
    r: File name too long
    Error occurred during initialization of VM
    Failure when starting JFR on_create_vm_3