search

pgxn / pgxn-api

Maintain and serve a REST API to search PGXN mirrors
http://pgxn.org/
15 stars 2 forks source link

Indexer Confused by Symbolic Links #24

Closed theory closed 3 years ago

theory commented 8 years ago

The recent release of pg_acl v0.1.0 triggered an error in the indexer. It chokes on pg_acl-0.1.0/test/pgxntool:

binmode() on closed filehandle GEN63 at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/5.14.2/x86_64-linux/IO/File.pm line 200.
at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/5.14.2/x86_64-linux/IO/File.pm line 200.
    IO::File::binmode(IO::File=GLOB(0x2f14de0)) called at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Archive/Zip.pm line 373
    Archive::Zip::_binmode(IO::File=GLOB(0x2f14de0)) called at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Archive/Zip/Member.pm line 1021
    Archive::Zip::Member::extractToFileHandle(Archive::Zip::ZipFileMember=HASH(0x2f0d9e0), IO::File=GLOB(0x2f14de0)) called at /var/virtuals/pgxn/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Archive/Zip/Member.pm line 489
    Archive::Zip::Member::extractToFileNamed(Archive::Zip::ZipFileMember=HASH(0x2f0d9e0), "www/src/pg_acl/pg_acl-0.1.0/test/pgxntool") called at lib/PGXN/API/Sync.pm line 200
    PGXN::API::Sync::unzip(PGXN::API::Sync=HASH(0x1466118), "dist/pg_acl/0.1.0/pg_acl-0.1.0.zip", HASH(0x2ef5bb0)) called at lib/PGXN/API/Sync.pm line 156
    PGXN::API::Sync::validate_distribution(PGXN::API::Sync=HASH(0x1466118), "dist/pg_acl/0.1.0/META.json") called at lib/PGXN/API/Sync.pm line 79
    PGXN::API::Sync::update_index(PGXN::API::Sync=HASH(0x1466118)) called at lib/PGXN/API/Sync.pm line 40
    PGXN::API::Sync::run(PGXN::API::Sync=HASH(0x1466118)) called at blib/script/pgxn_api_sync line 41

I downloaded that zip and unzipped it manually, and saw:

 unzip pg_acl-0.1.0.zip 
Archive:  pg_acl-0.1.0.zip
541ff97f51766f3d471471adab4baa8c44f13c46
   creating: pg_acl-0.1.0/
  inflating: pg_acl-0.1.0/.gitignore  
  inflating: pg_acl-0.1.0/LICENSE.md  
  inflating: pg_acl-0.1.0/META.in.json  
  inflating: pg_acl-0.1.0/META.json  
 extracting: pg_acl-0.1.0/Makefile   
  inflating: pg_acl-0.1.0/README.asc  
  inflating: pg_acl-0.1.0/pg_acl.control  
   creating: pg_acl-0.1.0/pgxntool/
 extracting: pg_acl-0.1.0/pgxntool/.gitignore  
  inflating: pg_acl-0.1.0/pgxntool/JSON.sh  
  inflating: pg_acl-0.1.0/pgxntool/JSON.sh.LICENCE  
  inflating: pg_acl-0.1.0/pgxntool/LICENSE  
  inflating: pg_acl-0.1.0/pgxntool/META.in.json  
  inflating: pg_acl-0.1.0/pgxntool/README.asc  
  inflating: pg_acl-0.1.0/pgxntool/_.gitignore  
  inflating: pg_acl-0.1.0/pgxntool/base.mk  
  inflating: pg_acl-0.1.0/pgxntool/build_meta.sh  
  inflating: pg_acl-0.1.0/pgxntool/meta.mk.sh  
  inflating: pg_acl-0.1.0/pgxntool/setup.sh  
   creating: pg_acl-0.1.0/pgxntool/test/
   creating: pg_acl-0.1.0/pgxntool/test/pgxntool/
  inflating: pg_acl-0.1.0/pgxntool/test/pgxntool/psql.sql  
  inflating: pg_acl-0.1.0/pgxntool/test/pgxntool/setup.sql  
  inflating: pg_acl-0.1.0/pgxntool/test/pgxntool/tap_setup.sql  
   creating: pg_acl-0.1.0/sql/
  inflating: pg_acl-0.1.0/sql/acl.sql  
   creating: pg_acl-0.1.0/test/
 extracting: pg_acl-0.1.0/test/deps.sql  
   creating: pg_acl-0.1.0/test/expected/
  inflating: pg_acl-0.1.0/test/expected/acl_type.out  
  inflating: pg_acl-0.1.0/test/expected/build.out  
  inflating: pg_acl-0.1.0/test/expected/compat.out  
  inflating: pg_acl-0.1.0/test/expected/rights.out  
    linking: pg_acl-0.1.0/test/pgxntool  -> ../pgxntool/test/pgxntool 
   creating: pg_acl-0.1.0/test/sql/
  inflating: pg_acl-0.1.0/test/sql/acl_type.sql  
  inflating: pg_acl-0.1.0/test/sql/build.sql  
  inflating: pg_acl-0.1.0/test/sql/compat.sql  
  inflating: pg_acl-0.1.0/test/sql/rights.sql  
finishing deferred symbolic links:
  pg_acl-0.1.0/test/pgxntool -> ../pgxntool/test/pgxntool

Note the last two lines: turns out that pg_acl-0.1.0/test/pgxntool is a symbolic link:

> ll test 
total 16
-rw-r--r--@ 1 david  staff    65B Jan  7 15:05 deps.sql
drwxr-xr-x@ 6 david  staff   204B Jan  7 15:05 expected/
lrwxr-xr-x  1 david  staff    25B Jan  8 09:10 pgxntool@ -> ../pgxntool/test/pgxntool
drwxr-xr-x@ 6 david  staff   204B Jan  7 15:05 sql/

Weirdly it seems to point to the wrong place; ../pgxntool/test/pgxntool does not exist, though ../pgxntool does.

Anyway, tweak the indexer to either resolve links to other files or directories in the zip file, or to ignore them altogether if they resolve to files outside of the zip file.

decibel commented 8 years ago

Hrm, it's possible that git archiving something with a submodule doesn't do the correct thing. Will investigate.

BTW, the symlink target certainly does exist in git...

decibel@decina:[16:41]~/git/pg_acl (master>)$ll pgxntool/test/pgxntool/
total 24
drwxr-x---  5 decibel  staff  170 Jan  3 15:27 ./
drwxr-x---  3 decibel  staff  102 Jan  2 16:43 ../
-rw-r-----  1 decibel  staff  172 Jan  2 16:43 psql.sql
-rw-r-----  1 decibel  staff  137 Jan  2 16:43 setup.sql
-rw-r-----  1 decibel  staff  198 Jan  3 15:27 tap_setup.sql
decibel@decina:[16:41]~/git/pg_acl (master>)$

Looking at your unzip output, it appears to be in the zip file correctly too...

   creating: pg_acl-0.1.0/pgxntool/test/pgxntool/
  inflating: pg_acl-0.1.0/pgxntool/test/pgxntool/psql.sql  
  inflating: pg_acl-0.1.0/pgxntool/test/pgxntool/setup.sql  
  inflating: pg_acl-0.1.0/pgxntool/test/pgxntool/tap_setup.sql
theory commented 8 years ago

Yeah, git archive ignores submodules by default. git-archive-all addresses that issue.

decibel commented 8 years ago

But it's putting the submodule in the zip. Or am I mis-understanding your output? My testing also shows it's there:

decibel@decina:[16:52]~/git/tmp$ll pg_acl-0.1.0/test/
total 16
drwxr-x---   6 decibel  staff  204 Jan  7 17:05 ./
drwxr-x---  12 decibel  staff  408 Jan  7 17:05 ../
-rw-r-----   1 decibel  staff   65 Jan  7 17:05 deps.sql
drwxr-x---   6 decibel  staff  204 Jan  7 17:05 expected/
lrwxr-x---   1 decibel  staff   25 Jan  8 16:52 pgxntool@ -> ../pgxntool/test/pgxntool
drwxr-x---   6 decibel  staff  204 Jan  7 17:05 sql/
decibel@decina:[16:53]~/git/tmp$ll pg_acl-0.1.0/test/pgxntool/
total 24
drwxr-x---  5 decibel  staff  170 Jan  7 17:05 ./
drwxr-x---  3 decibel  staff  102 Jan  7 17:05 ../
-rw-r-----  1 decibel  staff  172 Jan  7 17:05 psql.sql
-rw-r-----  1 decibel  staff  137 Jan  7 17:05 setup.sql
-rw-r-----  1 decibel  staff  198 Jan  7 17:05 tap_setup.sql
decibel@decina:[16:53]~/git/tmp$

So I suspect that whatever unzip module you're using doesn't know how to deffer symlink creation when unzipping...

theory commented 8 years ago

Oh, it does, that's why you can see the files in the browsable version. It's just the indexer that fails when it tries to pull out individual files. Will probably be an easy fix.

But in the short term, omitting symlinks will prevent the error and hopefully get your extension properly indexed (well, the docs, anyway; the full text won't be indexed until #2 is addressed).

theory commented 3 years ago

I reported this issue to Archive-Zip back in 2016, and just tried again and it no longer get the warning. Yay!