constraintAutomaton commented 4 months ago

Description

This PR create a debug/information actor to display the occupancy of the link queue. The information is outputted in the console via the logger. I've made a small library to parsed it into a JSON. A example of it is presented below.

A new implementation of #129

Features

Logging the:

The link pushed in the
The link pop in the queue
The timestamp of the links pushed or popped
The reachability criteria from which the link was acquire
The size of the link queue
The ratio of reachability criteria pushed and popped

Example outputs

{
    "SELECT ?messageId ?messageCreationDate ?messageContent WHERE {\\n  ?message <https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/vocabulary/hasCreator> <https://solidbench.linkeddatafragments.org/pods/00000000000000000933/profile/card#me>.\\n  ?message <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/vocabulary/Post>.\\n  ?message <https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/vocabulary/content> ?messageContent.\\n  ?message <https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/vocabulary/creationDate> ?messageCreationDate.\\n  ?message <https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/vocabulary/id> ?messageId.\\n}": {
        "pushEvents": [
            {
                "url": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/",
                "producedByActor": {
                    "name": "urn:comunica:default:extract-links/actors#predicates-solid",
                    "metadata": {
                        "predicates": [
                            "http://www.w3.org/ns/pim/space#storage"
                        ],
                        "matchingPredicate": "http://www.w3.org/ns/pim/space#storage",
                        "checkSubject": true
                    }
                },
                "timestamp": 1718631765370,
                "parent": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/profile/card",
                "queue": {
                    "size": 1,
                    "pushEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1
                    },
                    "popEvents": {}
                }
            },
            {
                "url": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/profile/",
                "producedByActor": {
                    "name": "urn:comunica:default:extract-links/actors#predicates-ldp",
                    "metadata": {
                        "predicates": [
                            "http://www.w3.org/ns/ldp#contains"
                        ],
                        "matchingPredicate": "http://www.w3.org/ns/ldp#contains",
                        "checkSubject": true
                    }
                },
                "timestamp": 1718631765472,
                "parent": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/",
                "queue": {
                    "size": 1,
                    "pushEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                        "urn:comunica:default:extract-links/actors#predicates-ldp": 1
                    },
                    "popEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1
                    }
                }
            },
            ...
            {
                "url": "https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/data/forum00000000755914244125",
                "producedByActor": {
                  "name": "urn:comunica:default:extract-links/actors#predicates-common",
                  "metadata": {
                    "predicates": [
                      "http://www.w3.org/2000/01/rdf-schema#seeAlso",
                      "http://www.w3.org/2002/07/owl##sameAs",
                      "http://xmlns.com/foaf/0.1/isPrimaryTopicOf"
                    ],
                    "matchingPredicate": "http://www.w3.org/2000/01/rdf-schema#seeAlso",
                    "checkSubject": false
                  }
                },
                "timestamp": 1718631766440,
                "parent": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/posts/2011-11-17",
                "queue": {
                  "size": 1,
                  "pushEvents": {
                    "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                    "urn:comunica:default:extract-links/actors#predicates-ldp": 91,
                    "urn:comunica:default:extract-links/actors#predicates-common": 32
                  },
                  "popEvents": {
                    "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                    "urn:comunica:default:extract-links/actors#predicates-ldp": 91,
                    "urn:comunica:default:extract-links/actors#predicates-common": 31
                  }
                }
              }
        ],
        "popEvents": [
            {
                "url": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/",
                "producedByActor": {
                    "name": "urn:comunica:default:extract-links/actors#predicates-solid",
                    "metadata": {
                        "predicates": [
                            "http://www.w3.org/ns/pim/space#storage"
                        ],
                        "matchingPredicate": "http://www.w3.org/ns/pim/space#storage",
                        "checkSubject": true
                    }
                },
                "timestamp": 1718631765370,
                "queue": {
                    "size": 0,
                    "pushEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1
                    },
                    "popEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1
                    }
                }
            },
            {
                "url": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/profile/",
                "producedByActor": {
                    "name": "urn:comunica:default:extract-links/actors#predicates-ldp",
                    "metadata": {
                        "predicates": [
                            "http://www.w3.org/ns/ldp#contains"
                        ],
                        "matchingPredicate": "http://www.w3.org/ns/ldp#contains",
                        "checkSubject": true
                    }
                },
                "timestamp": 1718631765473,
                "queue": {
                    "size": 4,
                    "pushEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                        "urn:comunica:default:extract-links/actors#predicates-ldp": 5
                    },
                    "popEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                        "urn:comunica:default:extract-links/actors#predicates-ldp": 1
                    }
                }
            },
            ...
            {
                "url": "https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/data/forum00000000755914244125",
                "producedByActor": {
                  "name": "urn:comunica:default:extract-links/actors#predicates-common",
                  "metadata": {
                    "predicates": [
                      "http://www.w3.org/2000/01/rdf-schema#seeAlso",
                      "http://www.w3.org/2002/07/owl##sameAs",
                      "http://xmlns.com/foaf/0.1/isPrimaryTopicOf"
                    ],
                    "matchingPredicate": "http://www.w3.org/2000/01/rdf-schema#seeAlso",
                    "checkSubject": false
                  }
                },
                "timestamp": 1718631766440,
                "queue": {
                  "size": 0,
                  "pushEvents": {
                    "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                    "urn:comunica:default:extract-links/actors#predicates-ldp": 91,
                    "urn:comunica:default:extract-links/actors#predicates-common": 32
                  },
                  "popEvents": {
                    "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                    "urn:comunica:default:extract-links/actors#predicates-ldp": 91,
                    "urn:comunica:default:extract-links/actors#predicates-common": 32
                  }
                }
              }
        ]
    }
}

Future works

Add information about the contribution the data source dereferenced.
Add information about the filters (shape index specific for the moment)

rubensworks commented 4 months ago

@RubenEschauzier, could you have a look at this first, given your previous work around inspecting the link queue?

constraintAutomaton commented 4 months ago

Just to be clear it's pretty much the same as #129 with the feedback taken into account. I started a new one because it felt easier to do so because the previous PR included also the reachability annotation which are now part of Comunica. (even thought this PR is a variable instead of hard coded field)

constraintAutomaton commented 4 months ago

And I don't know why the Solidbench test fail. this PR should have no effect on them.

rubensworks commented 4 months ago

And I don't know why the Solidbench test fail. this PR should have no effect on them.

Those tests can sporadically fail atm, so can be ignored. It's on my todo-list to look into that at some point.

constraintAutomaton commented 3 months ago

Hi @rubensworks and @RubenEschauzier , I wonder if you can review the PR when you have time. It is not of high importance.

constraintAutomaton commented 3 months ago

Could you show the new logger output after these changes are applied?

Yes, but the propose changes doesn't significantly change the data model (plural typos). And I've updated the proposed output in the description of the PR. I will inline it here.

{
    "SELECT ?messageId ?messageCreationDate ?messageContent WHERE {\\n  ?message <https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/vocabulary/hasCreator> <https://solidbench.linkeddatafragments.org/pods/00000000000000000933/profile/card#me>.\\n  ?message <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/vocabulary/Post>.\\n  ?message <https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/vocabulary/content> ?messageContent.\\n  ?message <https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/vocabulary/creationDate> ?messageCreationDate.\\n  ?message <https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/vocabulary/id> ?messageId.\\n}": {
        "pushEvents": [
            {
                "url": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/",
                "producedByActor": {
                    "name": "urn:comunica:default:extract-links/actors#predicates-solid",
                    "metadata": {
                        "predicates": [
                            "http://www.w3.org/ns/pim/space#storage"
                        ],
                        "matchingPredicate": "http://www.w3.org/ns/pim/space#storage",
                        "checkSubject": true
                    }
                },
                "timestamp": 1718631765370,
                "parent": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/profile/card",
                "queue": {
                    "size": 1,
                    "pushEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1
                    },
                    "popEvents": {}
                }
            },
            {
                "url": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/profile/",
                "producedByActor": {
                    "name": "urn:comunica:default:extract-links/actors#predicates-ldp",
                    "metadata": {
                        "predicates": [
                            "http://www.w3.org/ns/ldp#contains"
                        ],
                        "matchingPredicate": "http://www.w3.org/ns/ldp#contains",
                        "checkSubject": true
                    }
                },
                "timestamp": 1718631765472,
                "parent": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/",
                "queue": {
                    "size": 1,
                    "pushEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                        "urn:comunica:default:extract-links/actors#predicates-ldp": 1
                    },
                    "popEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1
                    }
                }
            },
            ...
            {
                "url": "https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/data/forum00000000755914244125",
                "producedByActor": {
                  "name": "urn:comunica:default:extract-links/actors#predicates-common",
                  "metadata": {
                    "predicates": [
                      "http://www.w3.org/2000/01/rdf-schema#seeAlso",
                      "http://www.w3.org/2002/07/owl##sameAs",
                      "http://xmlns.com/foaf/0.1/isPrimaryTopicOf"
                    ],
                    "matchingPredicate": "http://www.w3.org/2000/01/rdf-schema#seeAlso",
                    "checkSubject": false
                  }
                },
                "timestamp": 1718631766440,
                "parent": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/posts/2011-11-17",
                "queue": {
                  "size": 1,
                  "pushEvents": {
                    "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                    "urn:comunica:default:extract-links/actors#predicates-ldp": 91,
                    "urn:comunica:default:extract-links/actors#predicates-common": 32
                  },
                  "popEvents": {
                    "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                    "urn:comunica:default:extract-links/actors#predicates-ldp": 91,
                    "urn:comunica:default:extract-links/actors#predicates-common": 31
                  }
                }
              }
        ],
        "popEvents": [
            {
                "url": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/",
                "producedByActor": {
                    "name": "urn:comunica:default:extract-links/actors#predicates-solid",
                    "metadata": {
                        "predicates": [
                            "http://www.w3.org/ns/pim/space#storage"
                        ],
                        "matchingPredicate": "http://www.w3.org/ns/pim/space#storage",
                        "checkSubject": true
                    }
                },
                "timestamp": 1718631765370,
                "queue": {
                    "size": 0,
                    "pushEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1
                    },
                    "popEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1
                    }
                }
            },
            {
                "url": "https://solidbench.linkeddatafragments.org/pods/00000000000000000933/profile/",
                "producedByActor": {
                    "name": "urn:comunica:default:extract-links/actors#predicates-ldp",
                    "metadata": {
                        "predicates": [
                            "http://www.w3.org/ns/ldp#contains"
                        ],
                        "matchingPredicate": "http://www.w3.org/ns/ldp#contains",
                        "checkSubject": true
                    }
                },
                "timestamp": 1718631765473,
                "queue": {
                    "size": 4,
                    "pushEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                        "urn:comunica:default:extract-links/actors#predicates-ldp": 5
                    },
                    "popEvents": {
                        "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                        "urn:comunica:default:extract-links/actors#predicates-ldp": 1
                    }
                }
            },
            ...
            {
                "url": "https://solidbench.linkeddatafragments.org/www.ldbc.eu/ldbc_socialnet/1.0/data/forum00000000755914244125",
                "producedByActor": {
                  "name": "urn:comunica:default:extract-links/actors#predicates-common",
                  "metadata": {
                    "predicates": [
                      "http://www.w3.org/2000/01/rdf-schema#seeAlso",
                      "http://www.w3.org/2002/07/owl##sameAs",
                      "http://xmlns.com/foaf/0.1/isPrimaryTopicOf"
                    ],
                    "matchingPredicate": "http://www.w3.org/2000/01/rdf-schema#seeAlso",
                    "checkSubject": false
                  }
                },
                "timestamp": 1718631766440,
                "queue": {
                  "size": 0,
                  "pushEvents": {
                    "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                    "urn:comunica:default:extract-links/actors#predicates-ldp": 91,
                    "urn:comunica:default:extract-links/actors#predicates-common": 32
                  },
                  "popEvents": {
                    "urn:comunica:default:extract-links/actors#predicates-solid": 1,
                    "urn:comunica:default:extract-links/actors#predicates-ldp": 91,
                    "urn:comunica:default:extract-links/actors#predicates-common": 32
                  }
                }
              }
        ]
    }
}

rubensworks commented 3 months ago

@constraintAutomaton Can you share the new output after the changes you just did?

constraintAutomaton commented 3 months ago

@constraintAutomaton Can you share the new output after the changes you just did?

I've updated the description of the PR and the previous post related to it with the latest output.

rubensworks commented 1 month ago

Super, thanks @constraintAutomaton!

comunica / comunica-feature-link-traversal

Link queue wrapper to log the link queue occupancy #136

Description

Features

Example outputs

Future works