Building dependencies for `carton install` fails

[tukusejssirs]

I have troubles building gedcom dependencies.

Platform info

os   : CentOS 7 x86_64
perl : 5.16.3
cpan : 1.64

:exclamation: Note that I have not tested it on clean CentOS 7, therefore there might be other packages that need to be installed, but currently I have no time to setup up a test OS.

What have I done

1. Clone the repo.

   git clone git@github.com:nigelhorne/gedcom.git ~/gedcom_util
   cd ~/gedcom_util

2. Install carton

   cpan Carton

3. I tried to run carton install, but it failed installing GD module:

   ! Installing the dependencies failed: Module 'GD' is not installed, Module 'Geo::Coder::Free' is not installed, Module 'Image::Resize' is not installed

4. So I examined the issue and I needed to install gd-devel, but until I nailed the issue, I have installed php-gd and perl-GD packages too which are probably not necessary, but I list them anyway.

   sudo yum -y install php-gd perl-GD gd-devel
   sudo systemctl restart httpd  # Restart needed only because of `php-gd`

5. I tried to run carton install again, which tried to install Geo::Coder::Free, but failed to download http://download.maxmind.com/download/worldcities/worldcitiespop.txt.gz and therefore failed installing Geo::Coder::Free module altogether:

   # some lines removed
   Downloading http://download.maxmind.com/download/worldcities/worldcitiespop.txt.gz
   Fetch failed! HTTP response: 404 Not Found [404 Not Found] at Makefile.PL line 38.
   Fetch failed! HTTP response: 404 [Not Found] at Makefile.PL line 38.
   Command failed:  at Makefile.PL line 38.
   Command failed:  at Makefile.PL line 38.
   Command failed: get: Access failed: 404 Not Found (/download/worldcities/worldcitiespop.txt.gz)
   at Makefile.PL line 38.
   Command failed: HTTP/1.1 404 Not Found
   # some lines removed
   at Makefile.PL line 38.
   Got a '404' from 'download.maxmind.com' expected '200' at Makefile.PL line 38.
   Got a '404' from 'download.maxmind.com' expected '200' at /home/ts/perl5/lib/perl5/File/Fetch.pm line 812.
   File::Fetch::_iosock_fetch('File::Fetch=HASH(0x17ad248)', 'to', '/home/ts/.cpanm/work/1571219763.9427/Geo-Coder-Free-0.12/worl...') called at /home/ts/perl5/lib/perl5/File/Fetch.pm line 507
   File::Fetch::fetch('File::Fetch=HASH(0x17ad248)', 'http://download.maxmind.com/download/worldcities/worldcitiesp...') called at Makefile.PL line 38
   -> N/A
   -> FAIL No MYMETA file is found after configure. Your toolchain is too old?
   -> FAIL Configure failed for Geo-Coder-Free-0.12. See /home/ts/.cpanm/work/1571219763.9427/build.log for details.
   -> FAIL Installing the dependencies failed: Module 'Geo::Coder::Free' is not installed
   -> FAIL Bailing out the installation for /home/ts/git/others/gedcom_util/.

6. So I tried to install Geo::Coder::Free free manually. It required and/or suggested to install some additional modules, so I’ve ended up with installation of the following modules:

# Note that CGI::Lingua failed 2/451 subtests without IP::Country
# But before installing IP::Country, I installed Geo::IP, but then CGI::Lingua complained that it `Can't find your GeoIP.dat file for Geo::IP`
cpan GD Geo::Coder::Free Image::Resize YAML File::Fetch CGI::Lingua DBD::SQLite IP::Country

# I have also added this (it was suggested at the time of installation of some modules)
cpan Log::Log4perl

7. Third try of running carton install: it fails to detect that Geo::Coder::Free is installed and tries to install it again, which fails (cf step 5). But now I noticed in the ~/.cpanm/work/[some_numbers]/build.log that it tries to install v0.12 (from here), but I have installed (using cpan Geo::Coder::Free) v0.22. So the issue is probably in the carton install script, but I am not a pro programmer and I have never used carton, so I have no idea where to look for the script.

8. As a last resort, I removed local/ folder, installed the modules from requirements.txt and re-run carton install and it worked!

9. But running carton exec ./gedcom produced the following warning (right at the beginning of the output) about App::Cpan version, but it is not true ascpan App:Cpanoutputs the curretly installed version asv1.675`.

WARNING: your version of App::Cpan is 1.5701 while we would expect at least 1.64 at /home/ts/perl5/bin/cpan line 11.

10. But the output of `carton exec ./gedcom` ends with the following error:

```bash
"cpan -i Gedcom Genealogy::Gedcom::Date Date::Parse Geo::Coder::List Geo::Coder::Free IPC::System::Simple Geo::Coder::Ovi CHI String::Compare" unexpectedly returned exit value 1 at (eval 100) line 13.
 at ./gedcom line 30
BEGIN failed--compilation aborted at ./gedcom line 36.

11. I tried to issue that cpan command and its output looks good to me:

Reading '/home/ts/.cpan/Metadata'
  Database was generated on Tue, 15 Oct 2019 15:17:03 GMT
Gedcom is up to date (1.20).
Genealogy::Gedcom::Date is up to date (2.10).
Date::Parse is up to date (2.30).
Geo::Coder::List is up to date (0.26).
Geo::Coder::Free is up to date (0.22).
IPC::System::Simple is up to date (1.25).
Geo::Coder::Ovi is up to date (0.03).
CHI is up to date (0.60).
String::Compare is up to date (undef).

I have no idea how to solve this problem.

[tukusejssirs]

12. I had a new idea: run the ./gedcom file! I required additional Perl modules …

cpan Class::Simple::Cached Class::Simple::Readonly::Cached \
  Geo::Coder::OpenCage Geo::Coder::DataScienceToolkit
# I could not install `Geo::Coder::PlaceFinder` via `cpan`,
# because it could not find the matching namespace; here’s the output
Reading '/home/ts/.cpan/Metadata'
  Database was generated on Tue, 15 Oct 2019 15:17:03 GMT
Could not expand [Geo::Coder::PlaceFinder]. Check the module name.
I can suggest names if you install one of Text::Levenshtein::XS, Text::Levenshtein::Damerau::XS, Text::Levenshtein, and Text::Levenshtein::Damerau::PP
and you provide the -x option on invocation.
Skipping Geo::Coder::PlaceFinder because I couldn\'t find a matching namespace.  # Note I added the backslash to correct the MD syntax highlighting :)
# But `cpanm` did the job
cpanm Geo::Coder::PlaceFinder

13. I re-run ./gedcom and this is the output. I think you should correct those subroutine redefinition warnings. ;)

Subroutine Geo::Coder::PlaceFinder::ua redefined at ./gedcom line 7779.
Subroutine Geo::Coder::OpenCage::ua redefined at ./gedcom line 7790.
Usage: ./gedcom [ -a ] [ -A ] [ -b ] [ -B book.pdf ] [ -c ] [ -C ] [ -d ] [ -D ] [ -f ] [ -F ] [ -G ] [ -h home-person-name ] [ -H [ -L ] [ -m month ] [ -y year ] ] [ -l ] [ -p person-to-print ] [ -s ] [ -S ] [ -t ] [ -T ] [ -w [ -W ] [ -g | -x ]] filename [ filename2 ]

14. So it looks like it is working. So I tried to process my .ged file (generated by GenoPro 3.0.1.4) using ./gedcom -dl. It create an .ged.index file, but also output some warnings. Here is a simplified output log.

/home/ts/git/ofcl/rodokmen/exporty/rodokmen.ged:12: GLOBAL is not a top level tag
/home/ts/git/ofcl/rodokmen/exporty/rodokmen.ged:56: GENOMAP is not a top level tag
# The following error was on multiple marriages.
/home/ts/git/ofcl/rodokmen/exporty/rodokmen.ged:3905: Marriage is not a top level tag
# The following error was on multiple pedigree links.
/home/ts/git/ofcl/rodokmen/exporty/rodokmen.ged:3948: PEDIGREELINK is not a top level tag
# The following error was on multiple pictures.
/home/ts/git/ofcl/rodokmen/exporty/rodokmen.ged:5138: Picture is not a top level tag
# The following error was on multiple places.
/home/ts/git/ofcl/rodokmen/exporty/rodokmen.ged:5171: PLAC is not a top level tag
# The following error was on multiple twins.
/home/ts/git/ofcl/rodokmen/exporty/rodokmen.ged:5236: Twin is not a top level tag
# The following error was on multiple labels.
/home/ts/git/ofcl/rodokmen/exporty/rodokmen.ged:5248: LABEL is not a top level tag

[tukusejssirs]

15. I’ve just noticed that in BEGIN section of ./gedcom, there is Geo::Coder::PlaceFinder installed as cpan F/FR/FRIFFIN/Geo-Coder-PlaceFinder-0.1.tar.gz. But it was not installed; I have issued that command and it failed:

Reading '/home/ts/.cpan/Metadata'
  Database was generated on Tue, 15 Oct 2019 15:17:03 GMT
Fetching with LWP:
http://mirrors.nic.cz/CPAN/authors/01mailrc.txt.gz
Reading '/home/ts/.cpan/sources/authors/01mailrc.txt.gz'
............................................................................DONE
Fetching with LWP:
http://mirrors.nic.cz/CPAN/modules/02packages.details.txt.gz
Reading '/home/ts/.cpan/sources/modules/02packages.details.txt.gz'
  Database was generated on Wed, 16 Oct 2019 16:17:03 GMT
............................................................................DONE
Fetching with LWP:
http://mirrors.nic.cz/CPAN/modules/03modlist.data.gz
Reading '/home/ts/.cpan/sources/modules/03modlist.data.gz'
DONE
Writing /home/ts/.cpan/Metadata
Checksum for /home/ts/.cpan/sources/authors/id/F/FR/FRIFFIN/Geo-Coder-PlaceFinder-0.1.tar.gz ok
Configuring F/FR/FRIFFIN/Geo-Coder-PlaceFinder-0.1.tar.gz with Makefile.PL
Checking if your kit is complete...
Looks good
Generating a Unix-style Makefile
Writing Makefile for Geo::Coder::PlaceFinder
Writing MYMETA.yml and MYMETA.json
  FRIFFIN/Geo-Coder-PlaceFinder-0.1.tar.gz
  /usr/bin/perl Makefile.PL -- OK
Running make for F/FR/FRIFFIN/Geo-Coder-PlaceFinder-0.1.tar.gz
cp PlaceFinder.pm blib/lib/Geo/Coder/PlaceFinder.pm
Manifying 1 pod document
  FRIFFIN/Geo-Coder-PlaceFinder-0.1.tar.gz
  /usr/bin/make -- OK
Running make test for FRIFFIN/Geo-Coder-PlaceFinder-0.1.tar.gz
PERL_DL_NONLAZY=1 "/usr/bin/perl" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/placefinder.t .. 1/3 
#   Failed test 'use Geo::Coder::PlaceFinder;'
#   at t/placefinder.t line 10.
#     Tried to use 'Geo::Coder::PlaceFinder'.
#     Error:  Can't locate Net/OAuth.pm in @INC (@INC contains: /home/ts/.cpan/build/Geo-Coder-PlaceFinder-0.1-1/blib/lib /home/ts/.cpan/build/Geo-Coder-PlaceFinder-0.1-1/blib/arch /home/ts/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5/5.16.3 /home/ts/perl5/lib/perl5/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5/5.16.3 /home/ts/perl5/lib/perl5/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5 /home/ts/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5/5.16.3 /home/ts/perl5/lib/perl5/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5 /home/ts/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5/5.16.3 /home/ts/perl5/lib/perl5/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5 /home/ts/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5/5.16.3 /home/ts/perl5/lib/perl5/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5 /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /home/ts/.cpan/build/Geo-Coder-PlaceFinder-0.1-1/blib/lib/Geo/Coder/PlaceFinder.pm line 11.
# BEGIN failed--compilation aborted at /home/ts/.cpan/build/Geo-Coder-PlaceFinder-0.1-1/blib/lib/Geo/Coder/PlaceFinder.pm line 11.
# Compilation failed in require at t/placefinder.t line 10.
# BEGIN failed--compilation aborted at t/placefinder.t line 10.
Can't locate object method "new" via package "Geo::Coder::PlaceFinder" at t/placefinder.t line 16.
# Looks like your test exited with 255 just after 2.
t/placefinder.t .. Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 2/3 subtests 

Test Summary Report
-------------------
t/placefinder.t (Wstat: 65280 Tests: 2 Failed: 1)
  Failed test:  1
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 3 tests but ran 2.
Files=1, Tests=2,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.11 cusr  0.02 csys =  0.15 CPU)
Result: FAIL
Failed 1/1 test programs. 1/2 subtests failed.
make: *** [test_dynamic] Error 255
  FRIFFIN/Geo-Coder-PlaceFinder-0.1.tar.gz
  /usr/bin/make test -- NOT OK
//hint// to see the cpan-testers results for installing this module, try:
  reports FRIFFIN/Geo-Coder-PlaceFinder-0.1.tar.gz

16. These modules from the BEGIN section were also not installed (but they were manually installed successfully): Class::Simple::Cached and Class::Simple::Readonly::Cached.

17. Running ./gedcom -dAwWl name.ged was (kind of) successful, but it looks like the script cannot process correctly letters with diacritics (in my .ged file, there are Slovak and Hungarian names; example of letters with diacritics: ľščťžýáíéúäôňďěŕĺöüűő). The output is sometimes capitalises the letter (if any) after the letter with diacritics (example: lóS instead of lós) or inserts a space before it (example: l ós instead of lós). Somtimes it removes the space between name and surname. From what I’ve seen, these are the ‘rules’ (letters in the group at least once follow that particular rule):

• correctly output: éčĽíáúó
• capitalise following letter after these: óáťšžö
• insert space before these: óťš

There are also some errors/warnings in the output:

• Can't parse the name and Can't determine name of this person (the name contains diacritics);
• unknown location (I have some valid Slovak and Hungarian locations in the .ged file);
• FAMS fam00059 does not reference INDI ind00171.;
• No result found in country table and Unknown death country (when using [village], okr. [district] syntax);
• Unknown person: Not connected to the tree (I know that in my .ged file, there is a person that is not connected to any other person to form a family, but there are many warnings like this);
• Unknown person: failed consistency check and Unknown person: couldn't extract a biography (What are these errors about? When a person does not have a biography, it should fail graciously);
• ./gedcom does not ignore labels (1 NAME LABEL // or 3 LABEL or 1 ISLABEL Y);

I think that pasing as UTF-8 would solve this issue.

Also, is there a valid way to provide a district (okres in Slovak) name? I prefer to provide the district name to village name because some villages in Slovakia have the name name, but they are located in different district. I don’t do this for towns and cities. The format I use is [village], okr. [district] (okr. is shortcut for okres). Sometimes, when I don’t know village name, I use okr. [district]. I would be also awesome if district name could be its shortcut (see this table for district names; note that Bratislava and Kosice cities are listed multiple times because of its parts; also note that some districts have more than one shortcut to use in car plates: only the first one should be used in other cases, e.g. BB for Banská Bystrica. Note that I don’t want to use region names (in Slovak kraj; there are 8 regions in Slovakia).

Also note that I use ? for unknown mainly names and/or surnames, but also for unknown birth/death year. Again: is there a better way? I’d like it to be internationally understandable as I want the .ged file to be readable at least in English, Slovak and Hungarian.

[nigelhorne]

Yes, I think there is a bug in the prerequisite section for Geo::Coder::PlaceFinder. I've seen that before. Net::OAuth, Class::Simple::Cached and Class::Simple::Readonly::Cached are already loaded at line 34 in the latest version. Can you ensure you have that, please?

I'd need an example Slovak and/or Hungarian name to be sure of a fix that I could write. I don't have such people in my tree, but I'll take a look.

[tukusejssirs]

(1) I tried the new version using the following commands:

git clone git@github.com:nigelhorne/gedcom.git ~/gedcom_util_2
cd ~/gedcom_util_2
cpan Carton
carton install

carton install fails. Note that I did not test it with clean OS. Currently have no time for that. Here is the relevant output:

! Configure failed for Geo-Coder-Free-0.12. See /home/ts/.cpanm/work/1571253524.30523/build.log for details.
! Installing the dependencies failed: Module 'Geo::Coder::Free' is not installed
! Bailing out the installation for /home/ts/git/others/gcu/.
174 distributions installed
Installing modules failed

From the build.log:

# Geo-Coder-Free-0.12
Downloading http://download.maxmind.com/download/worldcities/worldcitiespop.txt.gz
Fetch failed! HTTP response: 404 Not Found [404 Not Found] at Makefile.PL line 38.
Fetch failed! HTTP response: 404 [Not Found] at Makefile.PL line 38.
# ...
-> FAIL No MYMETA file is found after configure. Your toolchain is too old?
-> FAIL Configure failed for Geo-Coder-Free-0.12. See /home/ts/.cpanm/work/1571253524.30523/build.log for details.

(2) Here are some name and surname examples. I would include some more for different languages because of additional letters with diacritics (like èë in French). Anyway, I think it is sufficiet to test with arbitrary names like ľščťžýáíéúäôňďěŕĺöüűőëèàźńùìòś and check if the encoding is correct.

• Táňa
• Mrkvička
• Mária
• Lőrincz
• Gaštan
• Ľubomír
• Ťažký
• Elemér
• Kôl

[nigelhorne]

That message about Geo::IP.dat not being found looks worse than it is. I've changed it to a warning, you can ignore it.

[nigelhorne]

You shouldn't need to use carton, since gedcom now installs the modules on first run. I've removed all references to it. Sorry about your wasted time.

[nigelhorne]

Please direct gedcom parsing messages such as

/home/ts/git/ofcl/rodokmen/exporty/rodokmen.ged:12: GLOBAL is not a top level tag

to the author of the gedcom module, at https://github.com/pjcj/Gedcom.pm. He can explain the messages far better than I.

[nigelhorne]

I use https://metacpan.org/pod/Lingua::EN::NameParse to parse names. I am not aware of a module that will do the same thing for other languages. I've looked around and found nothing. Are you aware of such a module?

[nigelhorne]

The "Can't parse the name" is actually a warning, I'll change the text to make it obvious what it's doing.

[tukusejssirs]

You shouldn't need to use carton, since gedcom now installs the modules on first run. I've removed all references to it. Sorry about your wasted time.

Okay, so all I should do is run ./gedcom and it should install all the dependences. Currently I cloned the repo again, run ./gedcom and it failed to install Date::ICal.

Can't locate Date/ICal.pm in @INC (@INC contains: /home/ts/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5/5.16.3 /home/ts/perl5/lib/perl5/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5 /home/ts/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5/5.16.3 /home/ts/perl5/lib/perl5/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5 /home/ts/perl5/lib/perl5/5.16.3/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5/5.16.3 /home/ts/perl5/lib/perl5/x86_64-linux-thread-multi /home/ts/perl5/lib/perl5 /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at ./gedcom line 47.
BEGIN failed--compilation aborted at ./gedcom line 47.

But after I have manually installed it (using cpan Date::ICal), it ran successfully. So, somehow the BEGIN section fails.

Are you aware of such a module?

No, I am not aware of any. However, Lingua::EN::NameParse has the following notes in its perlpod docs:

FUTURE DIRECTIONS

Define grammar for other languages. Hopefully, all that would be needed is to specify a new module with its own grammar, and inherit all the existing methods. I don't have the knowledge of the naming conventions for non-english languages.

BUGS

Names with accented characters (acute, circumfelx etc) will not be parsed correctly. A work around is to replace the character class [a-z] with \w in the appropriate rules in the grammar tree, but this could lower the accuracy of names based purely on ASCII text.

So, I think for now it would be good enough to use that workaround, but it would be nice (if it is possible) to re-replace the names with their original spelling after parsing, that is:

• remove the diacritics (Mária → Maria),
• parse the names as usual,
• replace the parsed names with their original form (Maria → Mária).

However, it would be much better to implement Lingua::SK::NameParse as it is written in the _Future directions. I’d like to contact Kim Ryan (the dev of Lingua::En::NameParse) if he is interested. Although I can code in Perl a bit, I am not a pro programmer. I could mainly assist in the liguist/algorithm part. Are you willing to help with the coding of this parser? Or you are busy enough with other stuff? :)

[nigelhorne]

I don't know why Date::ICal isn't installed, since it has this line:

system('cpan -i LWP::UserAgent::Cached Class::Simple::Cached Class::Simple::Readonly::Cached Data::ICal Data::ICal::Entry::Event Date::ICal');

[tukusejssirs]

Nor do I, but I had this problem before. :smile: I had to install following commands manually before I was able to run ./gedcom.

cpan Class::Simple::Cached Class::Simple::Readonly::Cached \
  Geo::Coder::OpenCage Geo::Coder::DataScienceToolkit
cpanm Geo::Coder::PlaceFinder

[tukusejssirs]

I’m gonna close this issue as it is resolved. I have created issues #99 and #100 based on conversation in this issue