Open ampli opened 8 years ago
The link-grammar.def file is not needed by Linux; its used only by Windows and MacOS, where it plays a vital role. Basically, it lists the functions that are publically visibile from a shared library.
I guess you are suggesting that we declare these functions as public API functions in various header files, but I really really don't want to do that, since they real are not means to be used by the public. A better solution would be to make the sat library be a STATIC libraries (not shared) and then link it directly into the LG library, if SAT compilation is enabled. That way, I believe we could remove most of the crud from the link-grammar.def file.
On Linux I just had to add these functions to linkgrammar.def - there was no need to define them as public PAI.
if so, isn't this linkgrammar.def addition enough also for Windows and MacOS?
In any case, I suggested above that we solve the need to add these in linkgrammar.def in another way: Put these functions as inline functions in ".h" files (which are included anyway in the places that now use these functions). Is this a better idea than a static SAT library?
Alternatively (instead of a static library), why note to just link the sat-solver files along with the rest of the link-grammar library files?
Well, yes, since we've configured the lib to use link-grammar.def, then yes, its used by Linux too.
If the functions are small then yes, maybe inline code in a header file might be OK, but is the cure worse than the disease?
Yes, we could link the SAT code straight into the LG library, but that would require writing some strange, convoluted, hard-to-debug autoconf code .. which, because its weird and hard-to-debug, I would not want to do. So, suggesting that it be a static lib gives you the same effect, but with less work and less confusion...
Here is a summary of problems that are still open after the latest fixes:
[()]
.
As far as I can see, the sat-solver code just doesn't encode these expressions, so their associated costs cannot be retrieved when a solution is found. I have an idea how to fix that, but I also have general questions on that. I will open a new issue.From all of these problems, I guess that fixing the cost problem is at the highest priority.
An additional thing related to point (4) above.
When using link-parser with !use_sat_solver=1
, if there is no complete linkage, currently it takes an additional time to get the linkparser> prompt after the message No complete linkages found. This is particularly noticeable with long sentences.
The reason is that by default !null=1, so if no complete linkage is found, link-parser tries to find a linkage with min_null_count=1 and max_null_count=sentence_length(). Then sat_parse()
tries again the same parsing (disregarding these setups), a thing that doubles the time to the the prompt again.
My suggested fix, until an encoding that allows parsing with nulls is implemented (I try to find out how to do that in an incremental way): 1. If min_null_count>0, issue a message and immediately return with no linkages. Something like:
if (opts->min_null_count > 0) {
// The sat solver doesn't support (yet) parsing with null words
if (opts->verbosity >= 1)
prt_error("Info: Cannot parse with null links yet (set !null=0 to turn off this message)\n");
sent->num_valid_linkages = 0;
sent->num_linkages_post_processed = 0;
return 0;
}
2. If min_null_count==0 but max_parse_count>0 (never happens with link-parser), try to parse anyway but if no complete linkage is found, also issue the said message.
I will send a pull request that implements that, but if you think it should be done in another way I can of course change it.
I tried LG with the following libraries:
Here is a summary. It turns out that my previous finding that Minisat 2.2+ performs the same as the current Minisat version doesn't hold for en/4.0.fixes-long.batch.
The tests were mainly with the default SAT-libraries parameters (I didn't test much with non-default ones, and the changes I tested didn't cause a speed-up).
On first glance it seemed that the comparison to the standard LG parser is not completely fair since the standard parser finds and sorts up to 1000 linkages per sentence even when running in batch mode, when the SAT parser only finds up to one valid linkage, since it currently finds linkages on demand only. However, from further tests it turned out that the overhead of fetching linkages by the standard parser is relatively negligible (for the SAT parser it is the main overhead).
In the tests described below, the SAT parser only finds the first valid linkage (if there is a one), while the standard parser produces up to 1000 valid ones (in both cases, the linkages themselves are not printed unless there is an error). Since producing linkages in the SAT solver is time-consuming, and "non-sane" linkages can be numerous for long sentences and must be skipped, adding "same alternative" constraints while encoding may be beneficial (not tested).
-timeout=100000
and -panic
(and no parsing with nulls got tried, since I used batch runs), the number of errors when running 4.0.fix-long.batch was not the same. This is strange. I don't know which one is more "correct", since the SAT encoding apparently has a problem with the disjunct costs.Conclusion: The currently used SAT library (0) has a severe problem with 2 example sentences (on one of them it "gets stuck"). Libraries (1) and (2) outperform it, and significantly so on long sentences. CryptoMiniSat outperforms all of them on long sentences, but has a very poor performance on shorter ones.
My recommendation is to replace the current Minisat version by Glucose-4.0 (3). BTW, CryptoMiniSat could be better if its relatively very bad results on short sentences could be fixed.
Also, I would first like to check the impact of improvements in the low-level part of generating the CNF (I don't understand yet the higher-level part of the LG linkage encoding).
would you still happen to have the raw numbers? Could you post them?
One reason the SAT work stagnated is because the original author found that he could not beat LG for normal-sized sentences. One could argue this was well-known -- SAT was first deployed in the chip-design industry, and it could not beat established tools for "ordinary"-sized IC's. When IC's got larger, SAT started winning by orders of magnitude: but you have to have big problems to seethe pay-off. So, also, it seams, in LG....
I've often thought of a "hybrid" mode: using the usual parser for sentences under 30-50 words, and then switching to SAT for anything longer.
Is the Glucose license compatible with LGPL?
The biggest reason to NOT use sat is that we need a parser that can handle conversations, where words dribble in one-at-a-time, or where several people are talking. For that viterbi would produce useful results, although it would be a lot SLOWER than the current LG.
If you are fishing for something to do that would be useful & cool, there are several things worth doing. One is partial-sentence support: implement a "virtual-left-right-wall" that captures the state (the unfulfilled disjuncts) in mid-sentence; then later, when the rest of the words come in, parsing resumes at this "virtual wall".
Another would be to handle quotations correctly, and/or multiple speakers: if you have two people saying things, can you untangle them into distinct streams? it would need probabilistic hints ...
would you still happen to have the raw numbers? Could you post them?
My relevant diary is attached here: fix-long-benchmark.txt
I've often thought of a "hybrid" mode: using the usual parser for sentences under 30-50 words, and then switching to SAT for anything longer.
I now try to optimize the low level clause generation by using PBlib. However, for really short sentences it is most probably not going to help since there is not much to optimize then in the low level.
Is the Glucose license compatible with LGPL?
It has a type of "free" license. I couldn't find the license online so I attache it here, taken from Glucose-4.0 extracted sources (renamed from LICENSE TO LICENSE.txt for the purpose of this attachment) : LICENSE.txt
One is partial-sentence support: implement a "virtual-left-right-wall" that captures the state (the unfulfilled disjuncts) in mid-sentence; then later, when the rest of the words come in, parsing resumes at this "virtual wall".
For that I will need some hints and help in how exactly approach this task, because I have already looked at it and couldn't conclude how it should be done.
Another would be to handle quotations correctly
I also started to investigate this problem. There is an article in my reading list that will hopefully give me useful insights: Punctuation in Quoted Speech. This article also discussed quoted text (p.63) and other interesting issues: ANALYSIS OF A FINITE-STATE GRAMMAR FOR PARSING AVIATION-SAFETY REPORTS.
if you have two people saying things, can you untangle them into distinct streams? it would need probabilistic hints ...
It seems to me that in order to do that in a good manner, the program needs to "understand" much more than what the current LG library can do.
[BTW, meanwhile I didn't forget the tokenizing problem and even the Hebrew dictionary.]
Could you create a directory called "diary" and place fix-long-benchmark.txt into that?
The LICENSE seems to be a BSD-type licence, so its compatible.
Also, I noticed that the russian tests have a very different behavior than english, with respect to timing.
For the mid-sentence "wall" think of it this way: if you cut the final parse of a sentence mid-way, you will cut some links, i.e. generate some connectors. On the first half of the sentence, the connectors all point right; the last half, they all point left. Thus, a cut sentence looks as if it could be "just some word" with connectors on it, i.e. a disjunct.
During parsing, there are more than one possibilities: so when you cut, there will be multiple possible disjuncts attached to the half-sentence ...
Could you create a directory called "diary" and place fix-long-benchmark.txt into that?
I will do that. I would also want to add the "ru" results.
For Russian there is a problem of mismatched alternatives ("insane linkages". Finding such "bogus" solutions is a significant overhead.
However, it is possible to SAT-encode the constraint of "select only words from the same alternative". This will remove the "insane" solutions. Adding constraints, by itself, may even make the SAT solving more efficient.
Well, careful about "constraints": SAT is a constraint-solver; in general, its best if you tell SAT what the constraint is, and let it solve it. Or at least, that is the principle. The practice, that's another matter, I guess.
Regarding adding constraints to eliminate "insane" linkages, I forgot to add:
Also, I noticed that the russian tests have a very different behavior than english, with respect to timing.
Here are my results: ru-banchmark.txt
Hmm. OK. Yes, switch got glucose, I guess. Sure, add constraints.
For sentences between 20-50 words, sometimes SAT is faster, sometimes the orig algo. One nutty idea is to start both, one in each thread, and first one to stop wins. Of course if the timeout is small, the orig algo will always win... the answer seems to be "use SAT if the sentence has a lot of words and the timeout is large" ....
the answer seems to be "use SAT if the sentence has a lot of words and the timeout is large" ....
This is indeed something that needs exploring.
Another thing I would like to explore is SAT parallelism. I already tried to set CryptoMiniSat for parallelism (but only with the default parameters) and the results where extremely bad - the elapse time was worse by much (I guess this depends on the problem to be solved, and there are problems in which SAT parallelism is more successful).
I still need to try the Glucose-4.0 parallelism.
ampli commented on Sep 19, 2015
- Different costs are shown in !disjuncts.
The main problem is expressions like that:
[()]
The current code assigns costs only to connectors, so I have no idea what to do with costs of null expressions.
Ah, good question. The costly-null is almost always used in the form A+ or [()]
which says "you can make the A link, or not. The A link costs nothing, but failing to make the A link costs 1.0". There are many places where this is used. The only one I can remember right now is the verb-wall link, which says "you had better link (WV) the head-verb to the wall, and if you don't it will cost you 2.0" which makes sense: if you have a sentence with a verb, you want to indicate it, as its more-or-less the most important part of the sentence.
Thus, as a corollary, expressions such as A+ or [()]
means that either you have to have the concept of a "null connector", or that costs must be a property of the disjunct, not the connector. Viz B+ & (A+ or [()])
must be expanded out by distributive property, as (B+ & A+) or [B+]
so that the null connector serves only to transfer the cost over to the grand-total disjunct.
I guess that if I would like to fix it, I will need to add a concept of a "null connector", because changing the SAT encoding code to have the cost on disjuncts seems to me too ambitious (and maybe unneeded).
The question is what the matching rules of a "null connector" are. Are they just "It always automagically matched, but with nothing"?
That seems like a dangerous idea. A simpler approach would be to always expand out any terms that have a null link in them, and bump the cost of the expanded form. Surely, the SAT solver already has some mechanism to expand null links, so you don't have to re-invent that. All you have to do is to propagate that cost onto some other .. any other .. connector in the expanded form. And since the SAT solver already has to deal with expressions like [C+ & (D+ or E+)]
there must be some way of shifting the cost over to somewhere else. (in the above, you can put the cost onto C+ or onto D+ AND E+ .. funny thing, how the or's turned into ands and v.v. in this last sentence!).
If I have an expression B+ & (A+ or [()])
, where should the cost be shifted to?
already answered that. B+ & (A+ or [()])
is identical to (B+ & A+) or [B+]
Update: By now, some of the remaining issues in the SAT parser are:
Hi Linas!
I mostly finished the full integration of the SAT solver. In addition to the previous fixes, that are listed in the comment in #187, I have fixed several more aspects of the SAT solver:
6. The maximum number of shown linkages is now !limit instead of the arbitrary 222.
Remained to fix: