metadata and user preferences

[romjerome]

Looking at current API specification, metadata might also look at some key values set on gramps desktop application ! Let me try to explain my comment on #48 .

web-api will generate new metadata endpoints. Some of them - like the researcher section - are set on the database; types are either hardcoded types or custom types (user) on gramps database. I suppose that metadata will retrieve existing data. This should be OK.

On the other hand some "endpoints" might be missing and are related to the loaded database, I am thinking on formats (date, name, place) or mapping (names). Having a date of creation and version of the database (like on current gramps xml format) might be also useful for handling multiple copies. I suppose that /metadata/summary will include at least date of creation and version? So, this should be OK too.

Sure, handling a gramps database could be done without values set by the user on gramps desktop application. We can ignore all keys related to interface, but like any locale options (LANG, LC*), users made some specific editions (which could be translated by user data or metadata). eg, grouping surnames is something difficult to guess or to handle (/metadata/surnames); custom (or localized) types too (/metadata/types_).

e.g., on my current database, I am looking at one lineage which often uses the patronymic HAENTZLER (it sounds like german but with french phonemes and letters). Distant cousins who lived in Paris (France) are sometimes registered as HANSLER, HENSLER. Distant cousins who lived in Berlin are rather registered as HÄNẞLER... Same lineage but multiple surnames or cultures! Spanish, Portuguese, Russian, Swedish/Icelandic/Norsk, etc. names might also have multiple "switch" according to individuals/ancestors/descendants! Sorting a list of surnames without a "user mapping key" or "name format support" might be limited under some non-english locales : a feature issue. As they are exported into the gramps XML file format, I suppose they are into the database too. So, web-API might get them too.

Place name format is maybe a little bit too complex as it seems an hybrid key for the gramps application but set for any family tree. So, maybe this could be ignored?

@DavidMStraub I have no idea about neither web API design nor database design! In the past, I just made some reflexions related to current gramps xml structure, where some duplicated sections might be merged. So, rather think on data handling issues like Attributes, Address object, Date object and user customizations.

About XPath on gramps xml, this custom listing should include all of them (gramps 4.2, 5.0, 5.1).

[DavidMStraub]

web-api will generate new metadata endpoints. Some of them - like the researcher section - are set on the database; types are either hardcoded types or custom types (user) on gramps database. I suppose that metadata will retrieve existing data. This should be OK.

Yes.

On the other hand some "endpoints" might be missing and are related to the loaded database, I am thinking on formats (date, name, place) or mapping (names). Having a date of creation and version of the database (like on current gramps xml format) might be also useful for handling multiple copies. I suppose that /metadata/summary will include at least date of creation and version? So, this should be OK too.

Version is there. Date of creation is not stored in the DB (or is it?)

Sure, handling a gramps database could be done without values set by the user on gramps desktop application. We can ignore all keys related to interface, but like any locale options (LANG, LC_*), users made some specific editions (which could be translated by user data or metadata). eg, grouping surnames is something difficult to guess or to handle (/metadata/surnames); custom (or localized) types too (/metadata/types).

Name groups could be added. They are stored in the DB and returned by db.get_name_group_keys(). This could simply be a separate endpoint. But I wouldn't add it under metadata because it is a different database table.

What you mean by custom types I have no idea. Custom event types are basically just strings that are stored along with the event. This string will be contained in the API response for that event. So I don't know what else you need.

e.g., on my current database, I am looking at one lineage which often uses the patronymic HAENTZLER (it sounds like german but with french phonemes and letters). Distant cousins who lived in Paris (France) are sometimes registered as HANSLER, HENSLER. Distant cousins who lived in Berlin are rather registered as HÄNẞLER... Same lineage but multiple surnames or cultures! Spanish, Portuguese, Russian, Swedish/Icelandic/Norsk, etc. names might also have multiple "switch" according to individuals/ancestors/descendants! Sorting a list of surnames without a "user mapping key" or "name format support" might be limited under some non-english locales : a feature issue. As they are exported into the gramps XML file format, I suppose they are into the database too. So, web-API might get them too.

Yes, name_group is in the DB and we could have a dedicated endpoint. But whether a given app built on the API actually implements this grouping is a different question ... I doubt any app will ever be as powerful as Gramps itself (but who knows...).

Place name format is maybe a little bit too complex as it seems an hybrid key for the gramps application but set for any family tree. So, maybe this could be ignored?

Not sure what you mean.

[romjerome]

Place name format is maybe a little bit too complex as it seems an hybrid key for the gramps application but set for any family tree. So, maybe this could be ignored?

Not sure what you mean.

I will try to explain with a test case. One would like to generate a heat map with surname frequency and place localisation (localization).

e.g., with 'Haentzler' surname (and variations): https://www.geneanet.org/nom-de-famille/HAENTZLER https://www.geneanet.org/nom-de-famille/HANSLER https://www.geneanet.org/nom-de-famille/HENSLER ... http://www.geopatronyme.com/cgi-bin/carte/nomcarte.cgi?nom=haentzler ... https://nvk.genealogy.net/map/1890:Hänßler,1996:Hänßler ... etc.

As he/she does not only want to query it locally, and maybe because he/she hopes/plans to find people who are looking at the same pair (surname, location) he/she made some experimentations by looking at current gramps data via a flat database (gramps xml, but you can transpose it by json or whatever SQLite DB).

He/she can quickly generate an interface for a cross search.

Compute/calculate number of entries matching the same surname or place name is also a simple task.

Now, he/she needs something more "smart" and complex than a simple raw parsing or a call for a serialization-like (get).

One would like to go further under the location hierarchy for a better reponse on its query. Also one knows that people may have some clues/details missing on some "global" databases (gov, GeoNames, etc.). So, something larger without the classical gedcom limitations on place names handling.

Looking at specific APIs related to top hierarchy level (country) he/she also got some mistakes.

Does it mean that errors will occur whatever databases and web API?

Maybe he/she needs to have a way, pattern, template, model for cultural issues and smart queries? e.g.,

• To match/display group of surnames.

``display_as | integer example: 0

group_as | string example:``

• To match (or localize via coordinates) common place names

To get {place-name format} might be used for having the same model for all place names into the database.

So, if one wants to query to get a pair like (surnames on place) and wants to share it with an other one using alternate place names (or place names on different periods), a smart reponse will return any possible match...

[romjerome]

What you mean by custom types I have no idea.

Not only custom types on primary objects (like Event), but also on secondary like attribute. Custom types are set on gramps' metadata DB table.

[DavidMStraub]

Not only custom types on primary objects (like Event), but also on secondary like attribute. Custom types are set on gramps' metadata DB table.

But we have a /metadata/types endpoint. Have you looked at the code?

[romjerome]

But we have a /metadata/types endpoint. Have you looked at the code?

        if datatype == "types":
            if args.get("type"):
                if args["type"] == "event_attribute":
                    result = db_handle.get_event_attribute_types()
                elif args["type"] == "event":
                    result = db_handle.get_event_types()
                elif args["type"] == "person_attribute":
                    result = db_handle.get_person_attribute_types()
                elif args["type"] == "family_attribute":
                    result = db_handle.get_family_attribute_types()
                elif args["type"] == "media_attribute":
                    result = db_handle.get_person_attribute_types()
                elif args["type"] == "family_relation":
                    result = db_handle.get_family_relation_types()
                elif args["type"] == "child_reference":
                   result = db_handle.get_child_reference_types()

@DavidMStraub do you mean I should remember that it is still incomplete?

Sorry, maybe I made a mix-up between attribute definition into gramps core app and object attribute on web-api (or attribute on any DB definition). By custom attribute types on DB tables, I mean something like:

/database/events/event[0]/attribute[0]/@ type /database/events/event[0]/objref[0]/attribute[0]/@ type => media /database/people/person[0]/eventref[0]/attribute[0]/@ type => event /database/people/person[0]/objref[0]/attribute[0]/@ type => media /database/people/person[0]/attribute[0]/@ type /database/families/family[0]/eventref[0]/attribute[0]/@ type => event /database/families/family[0]/objref[0]/attribute[0]/@ type = > media /database/families/family[0]/attribute[0]/@ type /database/citations/citation[0]/objref[0]/attribute[0]/@ type => media /database/sources/source[0]/objref[0]/attribute[0]/@ type => media /database/places/placeobj[0]/objref[0]/attribute[0]/@ type => media /database/objects/object[0]/attribute[0]/@ type => media

Sorry I cannot properly translate it into current web API documentation. Above lines are gramps XML paths against current web api endpoints. I hope this can make it clearer.

[DavidMStraub]

Sorry, but I simply don't understand what you're trying to tell me.

Why don't you give an example of an endpoint and a response and what it should return in your opinion, compared to what it currently returns.

[romjerome]

I just looked at code before testing endpoint... Do not expect any particular return, but current code could have issues with custom (user seizure) and hard-coded attribute types. e.g., around "media_attribute" and "event_attribute".

If I understand right : if args["type"] == "event_attribute": will return a list of all Attribute types assocated with Event instances in the database. https://www.gramps-project.org/docs/gen/gen_db.html?highlight=get_event_attribute_types#gramps.gen.db.base.DbReadBase.get_event_attribute_types if args["type"] == "event": will return a list of all event types in the database. https://www.gramps-project.org/docs/gen/gen_db.html?highlight=get_event_types#gramps.gen.db.base.DbReadBase.get_event_types if args["type"] == "person_attribute": will return a list of all Attribute types associated with Person instances in the database. https://www.gramps-project.org/docs/gen/gen_db.html?highlight=get_person_attribute_types#gramps.gen.db.base.DbReadBase.get_person_attribute_types if args["type"] == "family_attribute": will return a list of all Attribute types associated with Family instances in the database. https://www.gramps-project.org/docs/gen/gen_db.html?highlight=get_family_attribute_types#gramps.gen.db.base.DbReadBase.get_family_attribute_types if args["type"] == "media_attribute": will return a list of all Attribute types associated with Person instances in the database. https://www.gramps-project.org/docs/gen/gen_db.html?highlight=get_person_attribute_types#gramps.gen.db.base.DbReadBase.get_person_attribute_types

Is there a typo? Should not be something like

if args["type"] == "media_attribute": => result = db_handle.get_media_attribute_types() Return a list of all Attribute types assocated with Media and MediaRef instances in the database. https://www.gramps-project.org/docs/gen/gen_db.html?highlight=get_person_attribute_types#gramps.gen.db.generic.DbGeneric.get_media_attribute_types

Anyway, my primary issue was rather related to event and attribute type because in the past, some specific gedcom tags did not match gramps' event concept. So part of content moved to attribute types. A key/value was a good way for customization. These custom types were not always consistent on DB storage (metadata table). Also, we could generate an infinite loop (in theory) with eventref and objectref!

[DavidMStraub]

Is there a typo? Should not be something like if args["type"] == "media_attribute":

Yes this must be a typo. @cdhorn?

But apart from that, is there anything else you think is missing or can this issue be closed?

[cdhorn]

Yep, that needs fixing. I'll include in next PR, not worth something separate.

I hadn't checked, I hope example_gramps has at least one custom type of each in it.

[romjerome]

I hadn't checked, I hope example_gramps has at least one custom type of each in it.

I was not able to generate the theoretical infinite loop with attributes ( I suppose it is a good thing!), but I attached a sample with current possible editions into gramps application:

test_types.gramps.gz

Does it pass tests with current web-api version?

Some built-in attribute types are default keys (check lists), others were edited during a gramps session (stored as metadata). Sure, current design was sometimes inherited from old versions (e.g., gramps 1.x) or it needs to handle some specific data (e.g., gedcom importation) or it is just for the user choices (e.g., DNA stuff, occupation seizure, age or time definitions). I suppose this issue #49 can be closed.

Note, some posts around attribute types make me also think on an other test case around Form addon... i.e., close to my never finished experimental addon (which could be safety removed/hidden for next stable branches...). So, web-api might be used on a third-party application for editions before any import.

[cdhorn]

Some built-in attribute types are default keys (check lists), others were edited during a gramps session (stored as metadata).

Ideally any new application consuming the REST API should be able to query a type and get all the types, the custom and built in, and not be hard coding some subset of special cases. If the Gedcom ones are all hard coded in Gramps that is likely just the unfortunate influence of Gedcom on things early on. Opinions @DavidMStraub @Nick-Hall ?

[DavidMStraub]

Yes, exactly. The API doesn't really care - it can return all the custom types at the metadata endpoint and apart from that will always display the right type for objects, whether custom or not.

How to deal with that at an application level is a different question.

[DavidMStraub]

I think this is fixed by #61.