search

ShammyLevva / FTAnalyzer

Family Tree Analyzer - Finds hidden details in your family tree. Install at
http://www.ftanalyzer.com/install
Apache License 2.0
57 stars 23 forks source link

[BUG] Source list doesn't display correct # of citations #160

Closed nconway567 closed 4 years ago

nconway567 commented 4 years ago

Version 7.6.1.1 - Windows version. Using the Main Lists, Sources Tab. Several sources display 0 facts. Some display a # of facts. The counts are all wrong - even in cases where it shows 0. There are often source citations attached to the Person - not only facts such as birth, death etc. I just went and updated a source that FTA shows I had attached to three people. In reality, it was attached to a total of 15 people - but I had to manually search for them in roots magic.

Can this be fixed? It's sort of misleading.

ShammyLevva commented 4 years ago

I’ve not seen this myself can you create a small test file that demonstrates the issue so I can narrow down what’s happening please. If you upload the test GEDCOM file here or email it.

nconway567 commented 4 years ago

Attached is the Gedcom. There's 4 individuals. FTA lists 38 Sources. I specifically noticed one Ogle and Bothal (S325) is attached to King Edward III of England. It's attached as a Person source and a Marriage source (to Phillipa of Hainault), but FTA shows zero count. In fact, 23 out of 38 sources are listed with a zero count.

Roots Magic doesn't have a function to show who each source is attached to yet, so the only way to figure it out is to use your program or manually search - in other words - manually search and pray you find 'em all.

Thanks Nicole

From: Alexander Bisset notifications@github.com Sent: Wednesday, March 25, 2020 3:05 PM To: ShammyLevva/FTAnalyzer FTAnalyzer@noreply.github.com Cc: nconway567 nconway567@outlook.com; Author author@noreply.github.com Subject: Re: [ShammyLevva/FTAnalyzer] [BUG] Source list doesn't display correct # of citations (#160)

I’ve not seen this myself can you create a small test file that demonstrates the issue so I can narrow down what’s happening please. If you upload the test GEDCOM file here or email it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FShammyLevva%2FFTAnalyzer%2Fissues%2F160%23issuecomment-604028977&data=02%7C01%7C%7C3c07b2aec71543d13f3c08d7d0ef73ab%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637207599167333286&sdata=abW3eudaH%2FRvEK7qY8LtauCZrsQ4BOO2qL8RMOtnyaA%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM7ORSTEHEAVRPGE7N5Y2O3RJJIWVANCNFSM4LTQ72NA&data=02%7C01%7C%7C3c07b2aec71543d13f3c08d7d0ef73ab%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637207599167343282&sdata=fI8zbvPFH424KCovg7ejkXIPZwMMPTRaD%2F8PXzV%2FfLQ%3D&reserved=0.

ShammyLevva commented 4 years ago

Thanks I’ll have a look after work tomorrow.

ShammyLevva commented 4 years ago

I'm ready to have a look but see no GEDCOM. I suspect you attached it in an email (which gets stripped by GitHub) rather than uploading to the issue page.

nconway567 commented 4 years ago

Didnt realize that. Here's the file! test export FTA.zip

ShammyLevva commented 4 years ago

File received I'll have a look now.

ShammyLevva commented 4 years ago

Initial investigation suggests that all FACTS that have sources are working. RECORDS (ie: individuals and families) that have sources are not. This is because there isn't a "fact" to say a person or a family exists in the GEDCOM. So I think I'll need to create an "Individual" Fact and a "Family" fact to attach to records so that the sources can be tracked.

nconway567 commented 4 years ago

It also appears that FTA doesn't recognize "shared" facts. I'll upload another small gedcom - in this one, I have a shared fact named "kinship" that I use to tie a direct line together, such as someone who's a descendant of a king, magna carta descendant, etc. That fact is setup on most of the people in this gedcom - but it doesn't show up anywhere, even on the Stats page when it reads the file. I've also attached a 'test source' to this shared fact and that doesn't appear in the list of sources shared fact export.zip

. It seems like shared facts need to be added too?

ShammyLevva commented 4 years ago

It's that it doesn't recognise non standard fact types. Kinship is not a GEDCOM standard fact type so the likelihood is it's simply ignored. Some non standard ones are loaded but only if it uses the format of a custom fact type. I should be able to tell from the GEDCOM what's going on.

ShammyLevva commented 4 years ago

I don't have RootMagic so if I try to load it to FTM it's likely to strip out exactly what we are trying to see. Can you give me a specific example from the shared fact export file of what you mean please.

ShammyLevva commented 4 years ago

Hmm it seems that Roots Magic is making stuff up out of whole cloth - it's almost as bad as Legacy in that regard. It's created a completely non-standard format that isn't supported called _EVDEF.

These are random extra tags dumped into the the GEDCOM that don't follow the GEDCOM standard at all. In fact the format explicitly breaks the GEDCOM standard for creating additional tags.

Not sure how I feel about that other than initial anger. My gut feeling just now is well if Legacy and Roots Magic creates crap and tries to tie there users into their program I should be actively working to prevent that and NOT support such blatant lock-ins.

I've added the support for the standardised source records. I may NEVER add support for these deliberately standard breaking codes. I'm just not motivated to add these at present. Sorry.

ShammyLevva commented 4 years ago

Name sources added in v7.6.2

nconway567 commented 4 years ago

Ha! Why am I not surprised. Most of the bigger programs are using so much non standard stuff because the GEDCOM specs haven't been updated in years. Lots of programs are supporting shared Facts nowadays. I believe you can "share" any of their pre-programmed facts, not just the ones I created, because I've shared the residence fact and that's built in.

No worries on adding my custom built facts - I totally get your point of view.

From: Alexander Bisset notifications@github.com Sent: Friday, March 27, 2020 1:37 PM To: ShammyLevva/FTAnalyzer FTAnalyzer@noreply.github.com Cc: nconway567 nconway567@outlook.com; Author author@noreply.github.com Subject: Re: [ShammyLevva/FTAnalyzer] [BUG] Source list doesn't display correct # of citations (#160)

Hmm it seems that Roots Magic is making stuff up out of whole cloth - it's almost as bad as Legacy in that regard. It's created a completely non-standard format that isn't supported called _EVDEF.

These are random extra tags dumped into the the GEDCOM that don't follow the GEDCOM standard at all. In fact the format explicitly breaks the GEDCOM standard for creating additional tags.

Not sure how I feel about that other than initial anger. My gut feeling just now is well if Legacy and Roots Magic creates crap and tries to tie there users into their program I should be actively working to prevent that and NOT support such blatant lock-ins.

I've added the support for the standardised source records. I may NEVER add support for these deliberately standard breaking codes. I'm just not motivated to add these at present. Sorry.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FShammyLevva%2FFTAnalyzer%2Fissues%2F160%23issuecomment-605145772&data=02%7C01%7C%7C64128a18cd77448b3e8e08d7d2758fa2%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637209274662071179&sdata=gwMk7l4MD8HY3NWyZ9l76q1Syf7%2FoCMhkHEkF7aQaeI%3D&reserved=0, or unsubscribehttps://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM7ORSTD24DBYDYCGVBDQ4LRJTP6TANCNFSM4LTQ72NA&data=02%7C01%7C%7C64128a18cd77448b3e8e08d7d2758fa2%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637209274662081171&sdata=g545VxjRpqMZPXVMp66UdZUZo%2F%2BS00R4qpKLI8pV1A4%3D&reserved=0.

ShammyLevva commented 4 years ago

Most programs eg: FTM/Family Historian/GenoPro/Family Tree Builder/Reunion(Mac)/TreeView support custom facts and custom shared facts. ALL of those programs adhere to the GEDCOM standard and use a type of EVEN - to create a custom event.

Legacy uses _EVENT_DEFN and _PLACE_DEFN and RootsMagic it seems uses _EVDEF as a custom tag.

The problem is these don't appear to be simple variants of the standard EVEN tag. They seem to be defining types of events the adding these custom fact definitions. So the normal GEDCOM would be:

1 EVEN Earl of Chester 2 TYPE Noble Title 2 DATE BET 24 NOV 1312 AND 1327

Which is a custom fact type of Noble Title with a description of Earl of Chester. Where as the custom events seem to define the structure of the event as in...

0 _EVDEF Kinship 1 TYPE P 1 TITL Kinship 1 ABBR Kinship 1 SENT [NEED TO DEFINE SENTENCE: Kinship] 1 ROLE Witness 1 PLAC N 1 DATE N 1 DESC Y

This isn't a "Kinship" event this is a definition of how to store Kinship data. This means it's describing a NEW data structure not just a standard fact.

What I couldn't see in the file was where it applied this structure and how that was recorded. From the shared file who might have this fact applied and how does it look.

nconway567 commented 4 years ago

Oddly, Noble Title is a custom fact I created. RM's built in version is Title (Nobility). But the code starting with 0 _EVDEF Kinship looks a lot like the Fact definition screen. All the facts - built in or custom have those properties available. Where the difference may lie is in the Role - Principal or Witness. It may be that 'shared' facts are nothing more than a copy of a fact with a flag of Witness.

From: Alexander Bisset notifications@github.com Sent: Friday, March 27, 2020 1:56 PM To: ShammyLevva/FTAnalyzer FTAnalyzer@noreply.github.com Cc: nconway567 nconway567@outlook.com; Author author@noreply.github.com Subject: Re: [ShammyLevva/FTAnalyzer] [BUG] Source list doesn't display correct # of citations (#160)

Most programs eg: FTM/Family Historian/GenoPro/Family Tree Builder/Reunion(Mac)/TreeView support custom facts and custom shared facts. ALL of those programs adhere to the GEDCOM standard and use a type of EVEN - to create a custom event.

Legacy uses _EVENT_DEFN and _PLACE_DEFN and RootsMagic it seems uses _EVDEF as a custom tag.

The problem is these don't appear to be simple variants of the standard EVEN tag. They seem to be defining types of events the adding these custom fact definitions. So the normal GEDCOM would be:

1 EVEN Earl of Chester 2 TYPE Noble Title 2 DATE BET 24 NOV 1312 AND 1327

Which is a custom type of Noble Title with a description of Earl of Chester. Where as the custom events seem to define the structure of the event as in...

0 _EVDEF Kinship 1 TYPE P 1 TITL Kinship 1 ABBR Kinship 1 SENT [NEED TO DEFINE SENTENCE: Kinship] 1 ROLE Witness 1 PLAC N 1 DATE N 1 DESC Y

This isn't a "Kinship" event this is a definition of how to store Kinship data. This means it's describing a NEW data structure not just a standard fact.

What I couldn't see in the file was where it applied this structure and how that was recorded. From the shared file who might have this fact applied and how does it look.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FShammyLevva%2FFTAnalyzer%2Fissues%2F160%23issuecomment-605162848&data=02%7C01%7C%7C2a414229291847fc8e7308d7d2783e55%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637209286184082278&sdata=XfN%2FdEUbGdbTfhfT59OtUwrMkP5w%2FbMxCS7sb8MT3tY%3D&reserved=0, or unsubscribehttps://eur05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM7ORSQONV7D55K2LPPT4GDRJTSGTANCNFSM4LTQ72NA&data=02%7C01%7C%7C2a414229291847fc8e7308d7d2783e55%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637209286184082278&sdata=2rN6lxqQvf3T5rGZyThBbl9DiZNRQsOT13x2mGwEpec%3D&reserved=0.

ShammyLevva commented 4 years ago

So from the shared fact file you sent - what is an example of a shared fact - which Individuals (family group) has a specific fact so I can find it in the GEDCOM. I'm not yet aware of any specific example of a shared fact so I don't know what to look for.

nconway567 commented 4 years ago

Kinship is a shared fact, in this case the description reads Magna Carta Descendants. When I shared Kinship, I started with William Malet (b 1173) and copied it to all his direct line descendants until I reached Mark Hamill. Not all those ppl are in the gedcom - Mark Richard Hamill has it, as well as his father, gfather, ggfather and GGGfather.

In the RM database, there is in fact a Witness table, with structure of WitnessID, EventID, PersonID, WitnessOrder, Role, Sentence, Note (blob), so I would bet shared facts are really just a copy operation with keys pointing to the Originating person and each witness of that fact.

From: Alexander Bisset notifications@github.com Sent: Friday, March 27, 2020 2:11 PM To: ShammyLevva/FTAnalyzer FTAnalyzer@noreply.github.com Cc: nconway567 nconway567@outlook.com; Author author@noreply.github.com Subject: Re: [ShammyLevva/FTAnalyzer] [BUG] Source list doesn't display correct # of citations (#160)

So from the shared fact file you sent - what is an example of a shared fact - which Individuals (family group) has a specific fact so I can find it in the GEDCOM. I'm not yet aware of any specific example of a shared fact so I don't know what to look for.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FShammyLevva%2FFTAnalyzer%2Fissues%2F160%23issuecomment-605181216&data=02%7C01%7C%7C398cc164cd1a40e729d408d7d27a4765%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637209294925022507&sdata=gmXECOvqcoflyV2Q%2BVRSBXivW8Bw%2FnmYDJNR5DsRwkk%3D&reserved=0, or unsubscribehttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM7ORSURZ35R6LGNTEZKENLRJTT5HANCNFSM4LTQ72NA&data=02%7C01%7C%7C398cc164cd1a40e729d408d7d27a4765%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637209294925032515&sdata=fX%2B%2FAEgsLbkA5mh9gTN83bn4KQB%2F%2FZxVUq50tUZEegc%3D&reserved=0.

ShammyLevva commented 4 years ago

Mark Richard Hamill has...

0 @I69309@ INDI 1 NAME Mark Richard /Hamill/ 2 GIVN Mark Richard 2 SURN Hamill 1 SEX M 1 _UID 432377DD4DE94DDC9CFC438CE659496B7307 1 CHAN 2 DATE 9 MAR 2020 1 BIRT 2 DATE 25 SEP 1951 2 PLAC Oakland, Alameda, California, United States 1 EVEN Actor 2 TYPE Celebrity 1 EVEN King Henry II of England 2 TYPE Notable Relations 1 FAMC @F22872@

So he appears in a single Family as a child that family has...

0 @F22872@ FAM 1 HUSB @I69307@ 1 WIFE @I69308@ 1 CHIL @I69309@ 1 DIV 2 DATE JUN 1973 2 PLAC San Diego, California, United States 2 SOUR @S1222@ 3 PAGE https://familysearch.org/ark:/61903/1:1:VPBD-5DN; Virginia S Johnson and William T Hamill, Jun 1973 3 _TMPLT 4 FIELD 5 NAME BookID 4 FIELD 5 NAME Page 4 FIELD 5 NAME EntryNo 5 VALUE https://familysearch.org/ark:/61903/1:1:VPBD-5DN 4 FIELD 5 NAME PersonID 5 VALUE Virginia S Johnson and William T Hamill, Jun 1973 4 FIELD 5 NAME Date

So it doesn't look like the shared fact appears anywhere in the standard GEDCOM - can you confirm that the fact definitely appears in that small sample file?

ShammyLevva commented 4 years ago

He's got two custom facts one of type Actor and one of type Notable Relations. However no Kinship fact I can see.

nconway567 commented 4 years ago

Holy cow - Rootsmagic won't export out its own custom shared fact. It strips it out of the gedcom on export or when you import it. I just took that file I sent you and imported into a new RM database and the shared fact is not there. Bastards! Sorry to waste your time!!

[cid:e38b5791-47d4-41d8-8f54-0c94d34828a0]

From: Alexander Bisset notifications@github.com Sent: Friday, March 27, 2020 2:32 PM To: ShammyLevva/FTAnalyzer FTAnalyzer@noreply.github.com Cc: nconway567 nconway567@outlook.com; Author author@noreply.github.com Subject: Re: [ShammyLevva/FTAnalyzer] [BUG] Source list doesn't display correct # of citations (#160)

He's got two custom facts one of type Actor and one of type Notable Relations. However no Kinship fact I can see.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FShammyLevva%2FFTAnalyzer%2Fissues%2F160%23issuecomment-605202404&data=02%7C01%7C%7C6611719982084829a70408d7d27d3c36%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637209307624232774&sdata=BEWfvh1Ak94htxrHDdu27Vdbm%2FRjX0T61VI%2FFff2RvI%3D&reserved=0, or unsubscribehttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM7ORSQ67QHGTLVPZDZK26TRJTWMJANCNFSM4LTQ72NA&data=02%7C01%7C%7C6611719982084829a70408d7d27d3c36%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637209307624242782&sdata=6rFYTWClAJK2lnuMi5ANB4q%2BSfkTe3gQ6zqgRWBTFn4%3D&reserved=0.

ShammyLevva commented 4 years ago

Lol ok that's bizarre. Sounds like you need to raise that with RootsMagic that's a bug if ever I heard one.

No time wasted it was interesting. I'll release v7.6.2 soon to fix the original issue of lacking source records. May the force be with you :)

ShammyLevva commented 4 years ago

Sadly if RootsMagic isn't going to write out the data there's no Jedi mind tricks I can add to the code to extract the data by FORCE. So unfortunately this is not the fix you are looking for.

nconway567 commented 4 years ago

haha - I see what you did there.

I submitted a bug report. They are getting ready to release a new version so who knows if it's addressed already. Having the Person sources in FTA will be a big help. Thanks for that. Now if anyone knows of a utility that can fix all the random spaces in my notes that got broken when I imported that would be wonderful!

From: Alexander Bisset notifications@github.com Sent: Friday, March 27, 2020 2:48 PM To: ShammyLevva/FTAnalyzer FTAnalyzer@noreply.github.com Cc: nconway567 nconway567@outlook.com; Author author@noreply.github.com Subject: Re: [ShammyLevva/FTAnalyzer] [BUG] Source list doesn't display correct # of citations (#160)

Sadly if RootsMagic isn't going to write out the data there's no Jedi mind tricks I can add to the code to extract the data by FORCE. So unfortunately this is not the fix you are looking for.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FShammyLevva%2FFTAnalyzer%2Fissues%2F160%23issuecomment-605209489&data=02%7C01%7C%7Cd2c01a35cbb44c1d2b5508d7d27f7bc8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637209317279390748&sdata=yIE7A5W3um2BcQaP73DlIiDyqMJbfY6T%2BTK3IIpV81o%3D&reserved=0, or unsubscribehttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAM7ORSTOZA7GXLYFXHPEHPLRJTYI5ANCNFSM4LTQ72NA&data=02%7C01%7C%7Cd2c01a35cbb44c1d2b5508d7d27f7bc8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637209317279400761&sdata=YgtyAtJ%2FkeJFdYUsGpOgb5iS25YnpElF2QPzJx1X1Xs%3D&reserved=0.

ShammyLevva commented 4 years ago

Textpad is useful for that https://www.textpad.com/download/

It can do complex search and replaces to fix such things. You would need to copy and paste the text - fix it then copy and paste it back though.