When search returns many camliMember how to get details of each member in the result json?

[al-sabr]

Greetings all... so far so good on my way to get to know Perkeep.

I would like to know how is it possible to get deeper level of infos on a search request.

Exemple:

{
  "sort": "-created",
  "expression": "attr:type:product.category",
  "describe": {
    "depth": 1,
    "rules": [
      {
        "attrs": [
          "camliContent",
          "camliContentImage"
        ]
      },
      {
        "ifCamliNodeType": "foursquare.com:checkin",
        "attrs": [
          "foursquareVenuePermanode"
        ]
      },
      {
        "ifCamliNodeType": "foursquare.com:venue",
        "attrs": [
          "camliPath:photos"
        ],
        "rules": [
          {
            "attrs": [
              "camliPath:*"
            ]
          }
        ]
      }
    ]
  },
  "limit": 50
}

Returns this result:

{
  "blobs": [
    {
      "blob": "sha1-6450eb9dc391b087c0c3092235576553e55d78b8"
    }
  ],
  "description": {
    "meta": {
      "sha1-6450eb9dc391b087c0c3092235576553e55d78b8": {
        "blobRef": "sha1-6450eb9dc391b087c0c3092235576553e55d78b8",
        "camliType": "permanode",
        "size": 549,
        "permanode": {
          "attr": {
            "camliMember": [
              "sha1-ef2a991d7f148eb143f43d0402188d183341c103"
            ],
            "members": [],
            "title": [
              "categories"
            ],
            "type": [
              "product.category"
            ]
          },
          "modtime": "2018-01-11T18:34:41Z"
        }
      }
    }
  },
  "LocationArea": null
}

How to make it that the details on each member are listed in the same result?

[mpl]

just like the above example shows, you need to use Rules (https://perkeep.org/pkg/search#DescribeRequest) to recursively describe further down. So if instead of:

    "attrs": [
      "camliContent",
      "camliContentImage"
    ]

you do:

    "attrs": [
      "camliContent",
      "camliContentImage",
      "camliMember"
    ]

or even (if you don't care about camliContent and camliContentImage relationships):

    "attrs": [
      "camliMember"
    ]

it should do what you want.

and if you want to go even deeper, just add it again as another rules layer on the results. etc.

[al-sabr]

Ok It works now I've tried... my other question is: is there a way out of those result to ask Perkeep to build much more beautiful JSON object result like Category object which has many children which are Plugin object ? Is this possible or do I need to manually construct all myself.

{
  "type": "category",
  "parent": "sha1-parentpermanodeCategory",
  "numChildren": 6,
  "children": [
    {
      "pluginName": "Plugin",
      "author": "Author",
      "price": "2.99",
      "screenshots": [
        "sha1-imageblobreg",
        "sha1-imageblobreg",
        "sha1-imageblobreg"
      ]
    },
    {
      "pluginName": "Plugin",
      "author": "Author",
      "price": "2.99",
      "screenshots": [
        "sha1-imageblobreg",
        "sha1-imageblobreg",
        "sha1-imageblobreg"
      ]
    },
    {
      "pluginName": "Plugin",
      "author": "Author",
      "price": "2.99",
      "screenshots": [
        "sha1-imageblobreg",
        "sha1-imageblobreg",
        "sha1-imageblobreg"
      ]
    }
  ]
}

[al-sabr]

and if you want to go even deeper, just add it again as another rules layer on the results. etc.

I don't understand that part. Can you give an example please. How do I add it again as another rules layer?

[mpl]

The search handler repies to you with a https://perkeep.org/pkg/search#DescribedBlob, which results in the returned JSON you're seeing. Now, obviously, we're not going to add all possible fields to DescribedBlobs that clients could ever think of, like products categories or plugins, because the search handler is not the place for it.

What I think you want to do, is model your object as best as you can on permanodes with the tools you are given, which basically are: attributes, camliMember, camliPath, and camliContent. Then it is up to the clients to reconstruct a view of that model from the information they get from permanodes. As examples, you should look at how it's been done for the various importers. A tweet is a particular object with various parameters, and we store these parameters on a permanode. The search handler helps the clients (like the web UI) get all the permanodes about tweets, and then it is the job of the browser client to build a representation of a tweet with the information it gets from the permanodes.

Of course, as the search handler relies on the index for performance, when it makes sense, we index some of the information stored in permanodes to make searches more efficient, but note that there is no information specific about twitter (or any other importer) that is indexed right now. (https://perkeep.org/pkg/index/keys.go)

[al-sabr]

Ok so it is my responsibility to reconstruct manually from the received result. I'm no Go programmer so looking at the code is not going to help me since I don't know much about Go syntax.

can you answer also the https://github.com/perkeep/perkeep/issues/1035#issuecomment-357137037 thread? I don't quite understand well the recursive part of the rules.

thank you

[mpl]

About the rules, what I meant is: if you have rule1, and rule2, rule1 and rule2 are both applied to the set of results from search. If any of the rules "unveils" blobs that were not in the initial set, the rules are again applied to the new blobs. This repeats as long as new blobs are discovered by the rules.

So yeah, my sentence about "adding another layer" was confusing, sorry. It is more about planning the rules properly so that they recurse as deep as you want/need them.

[al-sabr]

Is it possible to get Descriptions inside of each pernamode which describes all the blobrefs (camliMember) instead of just listing the strings or Is it that i get only one big fat Description that I need to sort out myself? I find it very complicated to rebuild Category or Plugin based on the returned blobs or description.meta.[arrayofblobs]

The returned blobs are Category but the camliMember are Product

[mpl]

I'm not sure I understand your question. I think you would like the https://perkeep.org/pkg/search#SearchResult and the https://perkeep.org/pkg/search#DescribeResponse to be structured differently (which is very unlikely to happen), is that it? If yes, please explain how.

[al-sabr]

No I don't want the structure to change what I'm asking is there possibility with the search result to have inside of a blob which is per say a Category all its camliMember (Product) referenced to their actual Product blob with all the attributes to it.

Sort of cyclic dependency because right now only reference to their blob hash is listed and it sucks all the time to make the request to camli to get the details on the blobs.

In an another way does the search layer shows one to many or many to many kind of results.

I'm coming from Hibernate side and I like to have ValueObject right now the Data Modeling capabilities are not amazing or maybe I'm wrong. Do you get my point here ?

[mpl]

No, I don't understand. The JSON search response you get is just a JSON encoding of the types I mentioned above (SearchResult and DescribeResponse), nothing more. So I don't see how you could get the kind of relation you want (which i don't fully understand yet, a JSON example would help) in the JSON response without changing the SearchResult type structure.