Closed mammothb closed 4 years ago
There was a bug in my code. Please see the corrected version:
`
public bool LoadBigramDictionary(Stream corpusStream, int termIndex, int countIndex, char[] separatorChars = defaultSeparatorChars)
{
using (StreamReader sr = new StreamReader(corpusStream, System.Text.Encoding.UTF8, false))
{
String line;
int linePartsLength = (separatorChars == defaultSeparatorChars) ? 3 : 2;
//process a single line at a time only for memory efficiency
while ((line = sr.ReadLine()) != null)
{
string[] lineParts = line.Split(separatorChars);
if (lineParts.Length >= linePartsLength)
{
//if default (whitespace) is defined as separator take 2 term parts, otherwise take only one
string key = (separatorChars == defaultSeparatorChars) ? lineParts[termIndex] + " " + lineParts[termIndex + 1]: lineParts[termIndex];
//Int64 count;
if (Int64.TryParse(lineParts[countIndex], out Int64 count))
{
bigrams[key] = count;
if (count < bigramCountMin) bigramCountMin = count;
}
}
}
}
return true;
}
`
I see, so we change the number of line parts to check for depending on whether a custom separator was used. Thanks!
When I tried to load a dictionary file (using '$' as the separator) that looks like:
I wasn't able to add any terms to
bigrams
sinceline.Split(separatorChars)
split it into 2 parts. May I know how should we format the dictionary file content when loading it withLoadBigramDictionary
and a custom separator?