kbastani
commented
10 years ago Great question. They would be considered different distinct documents with the same training labels.
I originally went with an approach that extracted a set of sentences from a source text and ran the training algorithm sentence by sentence. I did this because repetition was important to training good models. This is no longer the case as I've made training the model focus more on quality of training examples.
For your idea to allow multiple documents to be sent during training, this works now, but the document being stored as a "Data" node, it has no unique identity, for instance a URL as an identifier of that document's text.
What I am going to do is to improve the data model to include an optional document identifier. This would be something you pass along during training:
{
"documents": [
{
"uri": "http://en.wikipedia.org/wiki/Interoperability",
"text": "Interoperability is the ability of making systems and organizations work together.",
"label": [
"Computing terminology",
"Telecommunications engineering",
"Interoperability",
"Product testing"
]
},
{
"uri": "http://en.wikipedia.org/wiki/Information_technology",
"text": "Information technology (IT) is the application of computers and telecommunications equipment to store, retrieve, transmit and manipulate data, often in the context of a business or other enterprise.",
"label": [
"Information technology",
"Media technology"
]
}
]
}Let me know what you think.
moooji
cicero19
Hi, this more of a question than an "issue": I noticed that during the training step I need to pass an array like:
{ "text": [ "Interoperability is the ability of making systems and organizations work together." ], "label": [ "Interoperability" ] }
to the endpoint, but in all of your examples the array contains only one element. I am wondering what it would mean for the classifier when I pass several elements in the "text" array for example. Would they be considered different elements of the same document, or would it see them as two separate documents which have the same label?
Related to this and as some input: It would be great if it would be actually possible to pass several documents with the same label in "one go" during training. That would reduce the amount of http requests drastically in my case and probably speed up training with 100.000s of small documents.
Just an idea :)