Closed sandreae closed 11 months ago
It might only be logged multiple times, haven't actually looked yet.
It also appears that previously published blobs get re-materialised to the file system on publishing a new one:
[2023-10-05T13:10:52Z INFO aquadoggo::materializer::tasks::blob] Creating blob at path /tmp/.tmp5gR7YM/00201c05353d50697f25f414399a7d56921f30aa6f03acd3a402d1293ad7cf355c60
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00200f33683892f5b5baedd8d3195ea9b787c6fdb3cf598619d9a5e7b77f16a99cf8
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 1 tasks
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::worker] Sending materializer blob task with input <TaskInput -/0020f8d2d049e14a6babfdf11dd15ce3d81c1a87bcf74f089db6a7f7ebbc4fe511b8> to the task queue.
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::blob] Working on <TaskInput -/0020f8d2d049e14a6babfdf11dd15ce3d81c1a87bcf74f089db6a7f7ebbc4fe511b8>
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::worker] Sending materializer blob task with input <TaskInput -/0020b4c8d8a4e4c66a1fad724fde3318a7e897981335c5a690b8c5c9e52d517f0e75> to the task queue.
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::blob] Working on <TaskInput -/0020b4c8d8a4e4c66a1fad724fde3318a7e897981335c5a690b8c5c9e52d517f0e75>
[2023-10-05T13:10:52Z INFO aquadoggo::materializer::tasks::blob] Creating blob at path /tmp/.tmp5gR7YM/0020f8d2d049e14a6babfdf11dd15ce3d81c1a87bcf74f089db6a7f7ebbc4fe511b8
[2023-10-05T13:10:52Z INFO aquadoggo::materializer::tasks::blob] Creating blob at path /tmp/.tmp5gR7YM/0020b4c8d8a4e4c66a1fad724fde3318a7e897981335c5a690b8c5c9e52d517f0e75
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 002060ac9914ef72be934ea73f26054ce5d7f584cc9df13b003fa3832b82e5130165
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020dd6ba77d7ff5b581a837ed8cfd809db38cd04bbcaf89ca0409a0f094d0fb9bd1
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020dd6ba77d7ff5b581a837ed8cfd809db38cd04bbcaf89ca0409a0f094d0fb9bd1
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 002065b45eae4493f68353ee0444b6af659099ee47c5095f14f806b01be77cd0c982
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 002065b45eae4493f68353ee0444b6af659099ee47c5095f14f806b01be77cd0c982
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00203e2d1eb00a92484e84a66c358984ebf71b5c853b36a482f50c62c0c711b92821
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00203e2d1eb00a92484e84a66c358984ebf71b5c853b36a482f50c62c0c711b92821
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 1 tasks
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::worker] Duplicate materializer blob task already in progress, setting re-queue flag for task with input <TaskInput -/00201c05353d50697f25f414399a7d56921f30aa6f03acd3a402d1293ad7cf355c60> and not adding this task to the queue.
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::worker] Sending materializer blob task with input <TaskInput -/00201c05353d50697f25f414399a7d56921f30aa6f03acd3a402d1293ad7cf355c60> to the task queue.
[2023-10-05T13:10:52Z DEBUG aquadoggo::materializer::tasks::blob] Working on <TaskInput -/00201c05353d50697f25f414399a7d56921f30aa6f03acd3a402d1293ad7cf355c60>
[2023-10-05T13:10:52Z INFO aquadoggo::materializer::tasks::blob] Creating blob at path /tmp/.tmp5gR7YM/00201c05353d50697f25f414399a7d56921f30aa6f03acd3a402d1293ad7cf355c60
Seems to actually be creating the blob again:
Maybe the issue comes from us creating a blob whenever a blob_piece_v1
or blob_v1
operation is processed in a blob_task
. This causes duplicate creations when all blob pieces were inserted into the store before the tasks were started.
If so, a better flow might be:
blob_v1
operation => attempt to create the blobblob_piece_v1
operation => identify the parent blob and issue a new blob task for itWith this I believe task de-duplication would take care of things for us.
If so, a better flow might be:
This is very similar to how we handle parent relationships, sounds good!
Hmmm, I don't think the above diagnosis explains this behaviour actually. I get these logs when publishing a new blob with id 00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
. As you can see, actually all blobs are repeatedly re-materialized to the file system.
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020e9e26e248795e9812c6cbc845ebfc684e51b461d5ca6db307478ee20e9ba805d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020e9e26e248795e9812c6cbc845ebfc684e51b461d5ca6db307478ee20e9ba805d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020e9e26e248795e9812c6cbc845ebfc684e51b461d5ca6db307478ee20e9ba805d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020e9e26e248795e9812c6cbc845ebfc684e51b461d5ca6db307478ee20e9ba805d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020e9e26e248795e9812c6cbc845ebfc684e51b461d5ca6db307478ee20e9ba805d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020e9e26e248795e9812c6cbc845ebfc684e51b461d5ca6db307478ee20e9ba805d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020e9e26e248795e9812c6cbc845ebfc684e51b461d5ca6db307478ee20e9ba805d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020e9e26e248795e9812c6cbc845ebfc684e51b461d5ca6db307478ee20e9ba805d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020e9e26e248795e9812c6cbc845ebfc684e51b461d5ca6db307478ee20e9ba805d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020b1fce8916c1a9b5b9c197b39e32705fc4950d5c8c45727c7f82fd782865e5dfa
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00205f8edadea98824587129216785f9fa9b43548d63f8169cf2037836f4f650648d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020f46a3f945167e17cfb6307d5d8e459dbe2d5e8343abb47e320205447d5f4ca68
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/0020e9e26e248795e9812c6cbc845ebfc684e51b461d5ca6db307478ee20e9ba805d
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/002012b7e9b8b80b1cbcf752ab1734fda6436d669becfb6fb9fb8a3f6505b4e667cd
I/aquadoggo::materializ..( 1844): Creating blob at path /data/user/0/com.p2panda.meli/files/00208084e70d7b975650be74a07c8e9173eec6194b75a7626fc4269ec266614a21e1
Ah, ok, it's this:
We're issuing dependency tasks eagerly for all documents of the task inputs "parent" schema. For blob pieces this means dependency tasks, and then blob tasks, are issued for every blob every time a piece arrives at the node :sweat_smile: .
I actually can't remember the reason for doing this "eagerly". Thinking about it now, it seems enough to issue dependency tasks for only documents which actually do refer to the task inputs document (meaning we look at values in relation fields rather than the schema id when searching for "parent" documents).
I actually can't remember the reason for doing this "eagerly".
If I understand your point correctly I think we need to keep it that "eagerly" as you otherwise will arrive at missing out third-tier, late-arriving operations not triggering materialization of the documents. https://whimsical.com/materializer-DbV6FUs51pqhouKff7Q546 - It's a complex topic, so might need to be revisited with some time to be 100% sure.
Something similar to this:
Maybe the issue is that the blob task doesn't have a way to prevent materialization when it already exists?
See for example: https://github.com/p2panda/aquadoggo/blob/main/aquadoggo/src/materializer/tasks/reduce.rs#L132-L134
It's fine to trigger many tasks in our design, they should just cancel early when they detect that they don't need to do the work (again).
In case there's a misunderstanding, I'll explain the issue I think I see again, don't mean I'm defo right, just to avoid confusion.
It appears that we are issuing dependency tasks for all documents which follow an identified "parent schema". So in the case where we have ParentSchema
and ChildSchema
with the parent containing a pinned relation to the child, the following flow would occur:
ChildSchemaDocument
at view X
ChildSchema
documentsParentSchema
ParentSchema
documents (even if they don't relate to the current tasks document view)I think we only want to issue dependency tasks for each individual document ~which refers to the current documents' view in a pinned relation (list)~ CORRECTION: which contains an operation which refers to any view of the current document in a pinned relation (list) . As shown here in the diagram:
Maybe the issue is that the blob task doesn't have a way to prevent materialization when it already exists?
This would also be a nice improvement :+1:
I think we only want to issue dependency tasks for each individual document which refers to the current documents' view in a pinned relation (list). As shown here in the diagram:
I think you will miss out on pinned views not getting materialized if they've get treated out of order.
It's a complex topic, so might need to be revisited with some time to be 100% sure.
Yep, and we don't need to revisit it now. For the current issue we can solve with the blob task being aware of the fact it is duplicating work.
(didn't mean to close :sweat_smile: )
I think we only want to issue dependency tasks for each individual document which refers to the current documents' view in a pinned relation (list). As shown here in the diagram:
I think you will miss out on pinned views not getting materialized if they've get treated out of order.
Hmm, yeh, I see :thinking:
I think we only want to issue dependency tasks for each individual document which refers to the current documents' view in a pinned relation (list). As shown here in the diagram:
I think you will miss out on pinned views not getting materialized if they've get treated out of order.
Hmm, yeh, I see 🤔
I made a correction in my explanation above.
We have a test checking against what I'm concerned about: https://github.com/p2panda/aquadoggo/blob/3ac8b70d230cf44f29e06a686e1251e1a8df9f17/aquadoggo/src/materializer/tasks/dependency.rs#L727 - so maybe it's enough we play with your idea and see if the tests still pass? I honestly don't have the focus and brain to think it through right now (just woke up being sick) 😆
As of the solution, I still don't think it would be enough to make the selection more "smart", the blob tasks still need to be able to cancel themselves if they're doing redundant work, as it would be still possible that they might be triggered in other race-condition scenarios. I'd argue that this is the actual "fix" for this problem, the other might be an "optimization" (if it wouldn't break our solution to the late arriving operations problem).
Yeh cool, I agree, I'll raise a separate issue for this "optimization" then and implement slightly smarter blob tasks now which avoid doing heavy work.
Logs from publishing a multi-piece blob: