Problems with tutorial code

[Ulitochka]

Hello.

Thnak you for this library.

I try to reproduce this tutotrial https://www.oreilly.com/ideas/how-to-get-superior-text-processing-in-python-with-pynini But I have this ERROR:

`import pynini

input_string = "Do you have Camembert or Edam?" cheeses = ("Boursin", "Camembert", "Cheddar", "Edam", "Gruyere", "Ilchester", "Jarlsberg", "Red Leicester", "Stilton") output_string = "Do you have Camembert or Edam"

fst_target = pynini.string_map(cheeses) ltag = pynini.transducer("", "") rtag = pynini.transducer("", "") substitution = ltag + fst_target + rtag

chars = ([chr(i) for i in range(1, 91)] + ["\[", "\]", "\\"] + [chr(i) for i in range(94, 256)]) sigma_star = pynini.string_map(chars).closure() rewrite = pynini.cdrewrite(substitution, "", "", sigma_star)`

ERROR: CDRewriteRule::Compile: sigma must be an unweighted acceptor Traceback (most recent call last): File "/home/m.domrachev/repos/attribute_parser/interpreters/grammar_fst/substit_fst.py", line 14, in rewrite = pynini.cdrewrite(substitution, "", "", sigma_star) File "pynini.pyx", line 1023, in pynini.cdrewrite File "pynini.pyx", line 1075, in pynini.cdrewrite File "pywrapfst.pyx", line 1818, in pywrapfst._MutableFst._check_mutating_imethod pywrapfst.FstOpError: Operation failed

[Ulitochka]

If i run test files: ~/pynini$ python3 pynini_test.py ..ERROR: CDRewriteRule::Compile: lambda must be an unweighted acceptor ....ERROR: CDRewriteRule::Compile: rho must be an unweighted acceptor ...ERROR: CDRewriteRule::Compile: lambda must be an unweighted acceptor .ERROR: CDRewriteRule::Compile: rho must be an unweighted acceptor ............ERROR: Verify: FST destination state ID of arc at position 0 of state 0 is negative .ERROR: State ID -1 not valid .ERROR: State ID -1 not valid .ERROR: State ID -1 not valid .ERROR: State ID -1 not valid .ERROR: State ID -1 not valid .ERROR: State ID -1 not valid .ERROR: State ID -1 not valid ...............ERROR: GenericRegister::GetEntry: nonexistent.so: cannot open shared object file: No such file or directory ERROR: Unknown weight type: nonexistent ..ERROR: FST and weight with non-matching weight types passed to AddArc: tropical and log .ERROR: FST and weight with non-matching weight types passed to Determinize: tropical and log .ERROR: FST and weight with non-matching weight types passed to Disambiguate: tropical and log .ERROR: FST and weight with non-matching weight types passed to Prune: tropical and log .ERROR: FST and weight with non-matching weight types passed to RmEpsilon: tropical and log .ERROR: FST and weight with non-matching weight types passed to SetFinal: tropical and log ...ERROR: ReadFstClass: Can't open file: nonexistent .........ERROR: Cyclic dependencies present in replacement set ...............................ERROR: GenericRegister::GetEntry: nonexistent-arc.so: cannot open shared object file: No such file or directory ERROR: CreateFstClass: Unknown arc type: nonexistent .ERROR: GenericRegister::GetEntry: nonexistent-arc.so: cannot open shared object file: No such file or directory ERROR: CreateFstClass: Unknown arc type: nonexistent .ERROR: StrToWeight: Bad weight = "nonexistent", source = WeightClass, line = 0 .ERROR: StrToWeight: Bad weight = "nonexistent", source = WeightClass, line = 0 .ERROR: FST and weight with non-matching weight types passed to CompileSymbolString: log and log64 .ERROR: FST and weight with non-matching weight types passed to CompileSymbolString: tropical and log .ERROR: StringFstToOutputLabels: State has multiple outgoing arcs ..ERROR: FST and weight with non-matching weight types passed to CrossProduct: log64 and tropical .ERROR: BracketedStringToLabels: Unmatched ] .ERROR: Unmatched [

[Ulitochka]

I try to run docker from https://github.com/NLPbox/pynini-docker/blob/master/Dockerfile And I have similar errors.

[Ulitochka]

python3.5 openfst-1.7.2 pynini-2.0.7

[kylebgorman]

The code in that tutorial is slightly out of date due to some modernizations to the library since then. Here's a variant that should work (changes the way we escape special characters):

chars = [chr(i) for i in range(1, 91)] + [r"\[", r"\\", r"\]"] + [chr(i) for i in range(94, 256)]
sigma_star = pynini.union(*chars).closure()

And so on.

FYI: that you see logged ERROR messages from the test is not an error! Some of the tests deliberately cause errors and check to make sure that the proper exceptions are raised. It should say something like:

Ran 163 tests in 0.156s

OK

at the bottom. If you see that OK, everything worked.

[Ulitochka]

Thanks!

[speechchemistry]

At the time of writing this, there are a few additional changes I had to make to get this cheese string tagging tutorial to work. I'm adding them here in case it's helpful to others:

ltag = pynini.transducer("", "<cheese>")
rtag = pynini.transducer("", "</cheese>")

should be changed to

ltag = pynini.cross("", "<cheese>")
rtag = pynini.cross("", "</cheese>")

and

output = pynini.compose(input_string, rewrite).stringify()

should be changed to

output = pynini.compose(input_string, rewrite).string()

[kylebgorman]

Please understand that there's nothing I can do to edit that particular document.

We had a relationship with O'Reilly because they had initially expressed interest in publishing a book about Pynini, but our acquisitions editor retired suddenly and her replacement was totally uninterested, and we haven't done any content for them in many years now.

[mabajec]

Hi all, what would be the most useful pynini documentation for one who would like to learn more about its use and specifics?

[kylebgorman]

Hi all, what would be the most useful pynini documentation for one who would like to learn more about its use and specifics?

The unit tests are never a bad place to start. Then, read the Pynini book itself. It sells for $45 USD (sorry, I don't have any control over the price, the publisher sold off the rights) and most academic libraries seem to have digital access to it.