This PR fixes ongoing issues with appflowy-indexer not updating the embeddings after some updates have been made.
This was related to following issue - apparently document_data.text_map stores all of the text fields, however not all of them are reachable from document root page down through blocks tree. Atm. more reliable source is to use document_data.blocks[id].data.delta.
This is to solve current issue at hand. The actual issue that needs to be solved is the fact, that text data is stored in two places: as delta in blocks and in text_map. In practice this may lead to issues in the futures (starting from memory issues since deltas are part of the yrs::Doc). If we need deltas we can generate them at any time from YText elements.
Below example for reference.
DocumentData {
page_id: "9gI6P0udyj",
blocks: {
"-NgRHs": Block {
id: "-NgRHs",
ty: "paragraph",
parent: "9gI6P0udyj",
children: "_5VN5h",
external_id: None,
external_type: None,
data: {
"delta": Array [
Object {
"insert": String("There's something else."),
},
],
},
},
"9gI6P0udyj": Block {
id: "9gI6P0udyj",
ty: "page",
parent: "",
children: "s2TuKx6qx8",
external_id: Some(
"w4y72_gQhV",
),
external_type: Some(
"text",
),
data: {},
},
"1zoNpl": Block {
id: "1zoNpl",
ty: "paragraph",
parent: "9gI6P0udyj",
children: "mBLndX",
external_id: None,
external_type: None,
data: {
"delta": Array [],
},
},
"u2T196cl8S": Block {
id: "u2T196cl8S",
ty: "heading",
parent: "9gI6P0udyj",
children: "eoGWzyAacI",
external_id: Some(
"d9tRqhVxtJ",
),
external_type: Some(
"text",
),
data: {
"level": Number(1),
"delta": Array [
Object {
"insert": String("Let's start from scratch"),
},
],
},
},
"YulIWe": Block {
id: "YulIWe",
ty: "paragraph",
parent: "9gI6P0udyj",
children: "LEpsct",
external_id: None,
external_type: None,
data: {
"delta": Array [
Object {
"insert": String("The\u{a0}"),
},
Object {
"attributes": Object {
"bold": Bool(true),
},
"insert": String("Peasants' Revolt"),
},
Object {
"insert": String(", also named\u{a0}"),
},
Object {
"attributes": Object {
"bold": Bool(true),
},
"insert": String("Wat Tyler's Rebellion"),
},
Object {
"insert": String("\u{a0}or the\u{a0}"),
},
Object {
"attributes": Object {
"bold": Bool(true),
},
"insert": String("Great Rising"),
},
Object {
"insert": String(", was a major uprising across large parts of England in 1381. The revolt had various causes, including the socio-economic and political tensions generated by the\u{a0}"),
},
Object {
"attributes": Object {
"href": String("https://en.wikipedia.org/wiki/Black_Death"),
},
"insert": String("Black Death"),
},
Object {
"insert": String("\u{a0}in the 1340s, the high taxes resulting from the conflict with France during the\u{a0}"),
},
Object {
"attributes": Object {
"href": String("https://en.wikipedia.org/wiki/Hundred_Years%27_War"),
},
"insert": String("Hundred Years' War"),
},
Object {
"insert": String(", and instability within the local leadership of London."),
},
],
},
},
"sx7BmF": Block {
id: "sx7BmF",
ty: "paragraph",
parent: "9gI6P0udyj",
children: "Asgar4",
external_id: None,
external_type: None,
data: {
"delta": Array [
Object {
"insert": String("The final trigger for the revolt was the intervention of a royal official, John Bampton, in\u{a0}"),
},
Object {
"attributes": Object {
"href": String("https://en.wikipedia.org/wiki/Essex"),
},
"insert": String("Essex"),
},
Object {
"insert": String("\u{a0}on 30 May 1381. His attempts to collect unpaid\u{a0}"),
},
Object {
"attributes": Object {
"href": String("https://en.wikipedia.org/wiki/Poll_taxes"),
},
"insert": String("poll taxes"),
},
Object {
"insert": String("\u{a0}in\u{a0}"),
},
Object {
"attributes": Object {
"href": String("https://en.wikipedia.org/wiki/Brentwood,_Essex"),
},
"insert": String("Brentwood"),
},
Object {
"insert": String("ended in a violent confrontation, which rapidly spread across the southeast of the country. A wide spectrum of rural society, including many local artisans and village officials, rose up in protest, burning court records and opening the local\u{a0}"),
},
Object {
"attributes": Object {
"href": String("https://en.wikipedia.org/wiki/Prison"),
},
"insert": String("prisons"),
},
Object {
"insert": String(". The rebels sought a reduction in taxation, an end to\u{a0}"),
},
Object {
"attributes": Object {
"href": String("https://en.wikipedia.org/wiki/Serfdom"),
},
"insert": String("serfdom"),
},
Object {
"insert": String(", and the removal of King\u{a0}"),
},
Object {
"attributes": Object {
"href": String("https://en.wikipedia.org/wiki/Richard_II"),
},
"insert": String("Richard II"),
},
Object {
"insert": String("'s senior officials and law courts."),
},
],
},
},
},
meta: DocumentMeta {
children_map: {
"mBLndX": [],
"eoGWzyAacI": [],
"s2TuKx6qx8": [
"u2T196cl8S",
"YulIWe",
"-NgRHs",
"sx7BmF",
"1zoNpl",
],
"Asgar4": [],
"LEpsct": [],
"_5VN5h": [],
},
text_map: Some(
{
"-DKHka": "[]",
"8-XTLr": "[{\"insert\":\"The final trigger for the revolt was the intervention of a royal official, John Bampton, in\u{a0}\"},{\"insert\":\"Essex\",\"attributes\":{\"href\":\"https://en.wikipedia.org/wiki/Essex\"}},{\"insert\":\"\u{a0}on 30 May 1381. His attempts to collect unpaid\u{a0}\"},{\"insert\":\"poll taxes\",\"attributes\":{\"href\":\"https://en.wikipedia.org/wiki/Poll_taxes\"}},{\"insert\":\"\u{a0}in\u{a0}\"},{\"insert\":\"Brentwood\",\"attributes\":{\"href\":\"https://en.wikipedia.org/wiki/Brentwood,_Essex\"}},{\"insert\":\"ended in a violent confrontation, which rapidly spread across the southeast of the country. A wide spectrum of rural society, including many local artisans and village officials, rose up in protest, burning court records and opening the local\u{a0}\"},{\"insert\":\"prisons\",\"attributes\":{\"href\":\"https://en.wikipedia.org/wiki/Prison\"}},{\"insert\":\". The rebels sought a reduction in taxation, an end to\u{a0}\"},{\"insert\":\"serfdom\",\"attributes\":{\"href\":\"https://en.wikipedia.org/wiki/Serfdom\"}},{\"insert\":\", and the removal of King\u{a0}\"},{\"insert\":\"Richard II\",\"attributes\":{\"href\":\"https://en.wikipedia.org/wiki/Richard_II\"}},{\"insert\":\"'s senior officials and law courts.\"}]",
"SMdptX": "[{\"insert\":\"The\u{a0}\"},{\"insert\":\"Peasants' Revolt\",\"attributes\":{\"bold\":true}},{\"insert\":\", also named\u{a0}\"},{\"insert\":\"Wat Tyler's Rebellion\",\"attributes\":{\"bold\":true}},{\"insert\":\"\u{a0}or the\u{a0}\"},{\"insert\":\"Great Rising\",\"attributes\":{\"bold\":true}},{\"insert\":\", was a major uprising across large parts of England in 1381. The revolt had various causes, including the socio-economic and political tensions generated by the\u{a0}\"},{\"insert\":\"Black Death\",\"attributes\":{\"href\":\"https://en.wikipedia.org/wiki/Black_Death\"}},{\"insert\":\"\u{a0}in the 1340s, the high taxes resulting from the conflict with France during the\u{a0}\"},{\"insert\":\"Hundred Years' War\",\"attributes\":{\"href\":\"https://en.wikipedia.org/wiki/Hundred_Years%27_War\"}},{\"insert\":\", and instability within the local leadership of London.\"}]",
"d9tRqhVxtJ": "[{\"insert\":\"Let's start from scratch\"}]",
"w4y72_gQhV": "[]",
"XcEu5Q": "[{\"insert\":\"There's something else.\"}]",
"_f5_RI": "[]",
},
),
},
}
This PR fixes ongoing issues with appflowy-indexer not updating the embeddings after some updates have been made.
This was related to following issue - apparently
document_data.text_mapstores all of the text fields, however not all of them are reachable from document root page down through blocks tree. Atm. more reliable source is to usedocument_data.blocks[id].data.delta.This is to solve current issue at hand. The actual issue that needs to be solved is the fact, that text data is stored in two places: as delta in blocks and in text_map. In practice this may lead to issues in the futures (starting from memory issues since deltas are part of the
yrs::Doc). If we need deltas we can generate them at any time from YText elements.Below example for reference.