search

FreshPorts / freshports

The website part of FreshPorts
http://www.freshports.org/
BSD 2-Clause "Simplified" License
68 stars 23 forks source link

Test dependencies are missing #411

Open yurivict opened 1 year ago

yurivict commented 1 year ago

The PHP code doesn't show them. I am not sure if the database contains them.

dlangille commented 1 year ago

I'll bet the code doesn't extract them.

dlangille commented 1 year ago

Here is an example:

[pkg01 dan ~/ports/head/sysutils/py-honcho] % make -V TEST_DEPENDS                               22:30:33
py39-mock>0:devel/py-mock@py39  py39-pytest>0:devel/py-pytest@py39 /usr/local/bin/python3.9:lang/python39
dlangille commented 1 year ago

Nope, the code does not extract that.

[dev-ingress01 dan ~/scripts] % grep -r TEST_DEPENDS *                                           22:31:03
[dev-ingress01 dan ~/scripts] % cd ~/modules                                                     22:31:05
[dev-ingress01 dan ~/modules] % grep -r TEST_DEPENDS *                                           22:31:11
grep: config.pm: Permission denied
[dev-ingress01 dan ~/modules] %                                                                  22:31:13
dlangille commented 1 year ago

It's pretty straight forward to modify the code and update the database for this. It takes about an hour. Then we need to add a one-time script to update all the ports. We also have to display the TEST_DEPENDS. And allow searching on that field.

I can do all the backend stuff soon. That's the scripts and modules directories you see above. That's in a repo here in my basement. It's not made the transition to git yet, but hopefully soon.

dlangille commented 1 year ago

dev.freshports.org has the code to read and store test_depends into the database. Once I have it working, I'll update your database.

dlangille commented 1 year ago

Now the front end at dev.freshports.org should display test_depends when some turn up.

yurivict commented 1 year ago

math/py-numpy has TEST_DEPENDS but dev.freshports.org doesn't show them: https://dev.freshports.org/math/py-numpy

dlangille commented 1 year ago

The current code will populate ports as they are updated (i.e. when a new commit is processed). A script will be written to populate test_depends of all ports.

In the meantime, these ports have updates and have values for test_depends:

freshports.devgit=# select id, element_pathname(element_id), test_depends from ports where test_depends is not null;
   id   |                  element_pathname                  |                                 test_depends                                 
--------+----------------------------------------------------+------------------------------------------------------------------------------
  49513 | /ports/head/devel/py-intbitset                     | py39-pytest>=7,1:devel/py-pytest@py39 /usr/local/bin/python3.9:lang/python39
  53715 | /ports/head/devel/folly                            | googletest>0:devel/googletest
 109140 | /ports/head/www/py-aiostream                       | /usr/local/bin/python3.9:lang/python39
 109139 | /ports/head/cad/py-gmsh                            | /usr/local/bin/python3.9:lang/python39
 109141 | /ports/head/cad/py-pygmsh                          | py39-pytest>=7,1:devel/py-pytest@py39 /usr/local/bin/python3.9:lang/python39
  49508 | /ports/head/devel/py-boolean.py                    | /usr/local/bin/python3.9:lang/python39
  49516 | /ports/head/textproc/py-pyahocorasick              | /usr/local/bin/python3.9:lang/python39
  97333 | /ports/head/www/py-dj-database-url                 | /usr/local/bin/python3.9:lang/python39
  97104 | /ports/head/textproc/py-mkdocs-material-extensions | /usr/local/bin/python3.9:lang/python39
  97103 | /ports/head/textproc/py-pymdown-extensions         | /usr/local/bin/python3.9:lang/python39
  97105 | /ports/head/textproc/py-mkdocs-material            | /usr/local/bin/python3.9:lang/python39
  36283 | /ports/head/deskutils/py-vdirsyncer                | /usr/local/bin/python3.9:lang/python39
(12 rows)

freshports.devgit=#

See https://dev.freshports.org/devel/py-intbitset/#requiredtest for example. I'm sure the others are also correct. I will commit my changes soon.

dlangille commented 1 year ago

I have updated your database instance:

freshports.org=# begin;
BEGIN
freshports.org=*# ALTER TABLE IF EXISTS public.ports
    ADD COLUMN test_depends text COLLATE pg_catalog."default";
ALTER TABLE
freshports.org=*# commit;
COMMIT
dlangille commented 1 year ago

And set one value for testing:

freshports.org=# begin;
BEGIN
freshports.org=*# update ports set test_depends = 'py39-pytest>=7,1:devel/py-pytest@py39 /usr/local/bin/python3.9:lang/python39' where id = 49513;
UPDATE 1
freshports.org=*# commit;
COMMIT
freshports.org=# select id, element_pathname(element_id), test_depends from ports where test_depends is not null;
  id   |        element_pathname        |                                 test_depends                                 
-------+--------------------------------+------------------------------------------------------------------------------
 49513 | /ports/head/devel/py-intbitset | py39-pytest>=7,1:devel/py-pytest@py39 /usr/local/bin/python3.9:lang/python39
(1 row)

freshports.org=#
dlangille commented 1 year ago

See https://r720-02.freshports.org/devel/py-intbitset/#dependencies

dlangille commented 1 year ago

If you pull my recent changes, and clear server-side cache, you should see test_depends at the above URL.

dlangille commented 1 year ago

Your database is being refreshed with test_depends values now.

tail -F /jails/ingress01/tmp/refresh-all-active-ports.log from the host will show progress.

yurivict commented 1 year ago

Thanks!

dlangille commented 1 year ago

The script finished. However, running it was premature. It didn't have the new code from dev.freshports.org.

dlangille commented 1 year ago

New code packaged and installed on host. Script running again.

dlangille commented 1 year ago

It's worse than I thought. The ingress01 jail does not have a freshports jail configured.

dlangille commented 1 year ago

Jail configured. Update script running again.

dlangille commented 1 year ago

From dev, this total seems very low.

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
    44
(1 row)

freshports.devgit=#

grep finds many more than that.

[dev-ingress01 dan /jails/freshports/usr/ports] % grep -rl TEST_DEPENDS *  | sort -u | wc -l
    2332
dlangille commented 1 year ago

Here is a short list for my future reference:

[dev-ingress01 dan /jails/freshports/usr/ports] % grep -rl TEST_DEPENDS *                                                                                                             15:27:22
CHANGES
Mk/bsd.java.mk
Mk/Uses/ghostscript.mk
Mk/Uses/magick.mk
Mk/Uses/python.mk
Mk/Uses/perl5.mk
Mk/Uses/pyqt.mk
Mk/Uses/pytest.mk
Mk/Uses/angr.mk
Mk/bsd.port.mk
archivers/p5-IO-Compress-Lzf/Makefile
archivers/p5-IO-Compress-Lzop/Makefile
archivers/py-libarchive-c/Makefile
archivers/py-borgbackup11/Makefile
archivers/py-lz4/Makefile
archivers/p5-Archive-SimpleExtractor/Makefile
archivers/p5-IO-Compress-Lzma/Makefile
archivers/R-cran-zip/Makefile
archivers/p5-Compress-Raw-Bzip2/Makefile
archivers/p5-Compress-Raw-Zlib/Makefile
archivers/p5-Archive-Libarchive/Makefile
archivers/p5-IO-Compress/Makefile
archivers/py-borgbackup/Makefile
archivers/gcpio/Makefile
archivers/p5-Compress-Raw-Lzma/Makefile
archivers/py-zstd/Makefile
archivers/gtar/Makefile
archivers/p5-Archive-Tar-Wrapper/Makefile
archivers/p5-Archive-Any-Lite/Makefile
archivers/p5-Archive-Extract-Libarchive/Makefile
archivers/py-xopen/Makefile
archivers/p5-Archive-Any/Makefile
astro/p5-Starlink-AST/Makefile
astro/py-skyfield-data/Makefile
astro/py-spktype21/Makefile
astro/py-spacetrack/Makefile
astro/p5-Astro-satpass/Makefile
astro/p5-Astro-Coords/Makefile
astro/p5-Astro-SIMBAD-Client/Makefile
astro/p5-Astro-Constants/Makefile
astro/py-jplephem/Makefile
astro/p5-Astro-PAL/Makefile
audio/libmysofa/Makefile
audio/py-opuslib/Makefile
audio/flacon/Makefile
audio/py-gtts/Makefile
audio/p5-Audio-MPD/Makefile
audio/py-aubio/Makefile
audio/moodbar/Makefile
audio/py-soundcloud/Makefile
audio/p5-Audio-Scan/Makefile
audio/sonivox/Makefile
audio/p5-Net-DAAP-DMAP/Makefile
benchmarks/py-locust/Makefile
benchmarks/libcelero/Makefile
biology/p5-Bio-NEXUS/Makefile
biology/p5-Bio-GFF3/Makefile
biology/p5-Bio-DB-NCBIHelper/Makefile
biology/bcftools/Makefile
biology/bedtools/Makefile
biology/spoa/Makefile
biology/stringtie/Makefile
biology/p5-Bio-DB-EMBL/Makefile
biology/p5-Bio-MAGETAB/Makefile
biology/biosoup/Makefile
biology/htslib/Makefile
biology/bioparser/Makefile
biology/p5-Bio-Phylo/Makefile
biology/metaeuk/Makefile
biology/p5-BioPerl/Makefile
biology/samtools/Makefile
biology/py-cutadapt/Makefile
biology/py-ont-fast5-api/Makefile
biology/unikmer/Makefile
cad/graywolf/Makefile
cad/py-ezdxf/Makefile
cad/xyce/Makefile
cad/nvc/Makefile
cad/py-gdspy/Makefile
cad/yosys/Makefile
cad/cascade-compiler/Makefile
comms/p5-Nexmo-SMS/Makefile
comms/py-fritzconnection/Makefile
comms/py-xmodem/Makefile
comms/py-esptool/Makefile
converters/p5-JSON-MaybeXS/Makefile
converters/osm2pgsql/Makefile
converters/p5-Sereal-Encoder/Makefile
converters/p5-Encode-IMAPUTF7/Makefile
converters/py-bencode.py/Makefile
converters/p5-Sereal/Makefile
converters/p5-JSON-Any/Makefile
converters/p5-Unicode-Stringprep/Makefile
converters/py-svglib/Makefile
converters/p5-Encode-Base32-Crockford/Makefile
converters/p5-Sereal-Decoder/Makefile
converters/p5-JSON-Tiny/Makefile
converters/py-text-unidecode/Makefile
converters/p5-Cpanel-JSON-XS/Makefile
converters/p5-Encode-Base32-GMP/Makefile
converters/p5-Encode-Base58-GMP/Makefile
converters/p5-Convert-BinHex/Makefile
databases/p5-Search-Xapian/Makefile
databases/p5-DBM-Deep/Makefile
databases/p5-Class-DBI-Plugin-Pager/Makefile
databases/p5-DBIx-Class-DateTime-Epoch/Makefile
databases/p5-CHI-Driver-Memcached/Makefile
databases/p5-DBIx-Abstract/Makefile
databases/py-fakeredis/Makefile
databases/py-pgcli/Makefile
databases/p5-DBIx-Class-Tree-NestedSet/Makefile
databases/p5-DBIx-Class-Candy/Makefile
databases/p5-DBIx-Class/Makefile
databases/p5-DBIx-Class-Schema-Config/Makefile
databases/p5-DBIx-Wrapper/Makefile
databases/p5-SQL-Abstract-Limit/Makefile
databases/py-python-binary-memcached/Makefile
databases/p5-DBICx-TestDatabase/Makefile
databases/py-sqlalchemy11/Makefile
databases/p5-SQL-Translator/Makefile
databases/p5-Class-DBI-Plugin-Iterator/Makefile
databases/p5-HTML-FormHandler-Model-DBIC/Makefile
databases/p5-App-Sqitch/Makefile
databases/p5-DBIx-Class-Storage-TxnEndHook/Makefile
databases/p5-Protocol-CassandraCQL/Makefile
databases/p5-DBIx-Class-DynamicDefault/Makefile
databases/p5-DBIx-XHTML_Table/Makefile
databases/cassandra3/Makefile
databases/ldb22/Makefile
databases/p5-Dancer2-Session-DBIC/Makefile
databases/p5-Rose-DB-Object/Makefile
databases/p5-Class-DBI-Plugin-RetrieveAll/Makefile
databases/p5-DBIx-Class-Schema-PopulateMore/Makefile
databases/p5-ORLite/Makefile
databases/p5-DBIx-Class-Schema-Loader/Makefile
databases/p5-Redis-Fast/Makefile
databases/pglogical/Makefile
databases/p5-DBIx-DBHResolver/Makefile
databases/p5-DBIx-Inspector/Makefile
databases/p5-DBIx-Class-Migration/Makefile
databases/p5-DBD-ODBC/Makefile
databases/py-sqlite-utils/Makefile
databases/p5-DBIx-Class-QueryProfiler/Makefile
databases/p5-DBIx-Skinny-Pager/Makefile
databases/p5-DBIx-QuickDB/Makefile
databases/p5-DBIx-Handler/Makefile
databases/p5-Test-mysqld/Makefile
databases/p5-DBIx-SearchBuilder/Makefile
databases/py-databases/Makefile
databases/p5-DBIx-Class-ResultSet-RecursiveUpdate/Makefile
databases/p5-DBIx-Tracer/Makefile
databases/p5-Dancer-Plugin-Database/Makefile
databases/py-marshmallow-sqlalchemy/Makefile
databases/py-cassandra-driver/Makefile
databases/p5-DBICx-Sugar/Makefile
databases/py-redis2/Makefile
databases/py-duckdb/Makefile
databases/p5-Test-Fixture-DBI/Makefile
databases/p5-SQL-Abstract/Makefile
databases/p5-BSON/Makefile
databases/p5-DBIx-Admin-CreateTable/Makefile
databases/p5-SQL-Abstract-More/Makefile
databases/p5-Cache-Memcached-Fast/Makefile
databases/py-sqlalchemy12/Makefile
databases/py-lmdb/Makefile
databases/p5-DBIx-Class-Helpers/Makefile
databases/p5-AnyEvent-Redis/Makefile
databases/p5-Class-DBI-Plugin-Type/Makefile
databases/p5-CHI-Driver-Redis/Makefile
databases/p5-Test-postgresql/Makefile
databases/py-Pyrseas/Makefile
databases/p5-DBIx-Class-TimeStamp/Makefile
databases/p5-BerkeleyDB/Makefile
databases/p5-DBIx-ContextualFetch/Makefile
databases/py-aesqlapius/Makefile
databases/p5-DBIx-Class-AsFdat/Makefile
databases/p5-DBIx-Class-BitField/Makefile
databases/p5-DBIx-Sunny/Makefile
databases/p5-Net-Async-CassandraCQL/Makefile
databases/p5-DBIx-Class-MooseColumns/Makefile
databases/leveldb/Makefile
databases/p5-DBIx-Class-Cursor-Cached/Makefile
databases/libmemcached/Makefile
databases/py-ormar/Makefile
databases/p5-Text-Query-SQL/Makefile
databases/p5-DBIx-Class-Fixtures/Makefile
databases/py-peewee_migrate/Makefile
databases/p5-DBIWrapper/Makefile
databases/py-pony/Makefile
databases/p5-Data-Page/Makefile
databases/p5-Search-Xapian12/Makefile
databases/p5-Amon2-DBI/Makefile
databases/p5-Class-DBI/Makefile
databases/p5-DBICx-TxnInsert/Makefile
databases/p5-SQL-NamedPlaceholder/Makefile
databases/p5-Interchange6-Schema/Makefile
databases/p5-AnyEvent-DBD-Pg/Makefile
databases/p5-DBIx-Skinny-InflateColumn-DateTime/Makefile
databases/p5-DBIx-Class-ResultSet-HashRef/Makefile
databases/p5-Class-DBI-DDL/Makefile
databases/p5-Teng/Makefile
databases/p5-DBD-mysql/Makefile
databases/p5-DBIx-Class-InflateColumn-IP/Makefile
databases/p5-RedisDB/Makefile
databases/py-sqlalchemy10/Makefile
databases/p5-DBIx-Class-Tree/Makefile
databases/py-alembic14/Makefile
databases/p5-DBD-Multi/Makefile
databases/p5-DBIx-Class-Loader/Makefile
databases/p5-Cache-Memcached-Managed/Makefile
databases/p5-Scope-Container-DBI/Makefile
databases/p5-Class-DBI-Plugin-DeepAbstractSearch/Makefile
databases/p5-Dancer2-Plugin-Database/Makefile
databases/p5-MongoDB/Makefile
databases/p5-DBIx-QueryLog/Makefile
databases/py-alembic/Makefile
databases/p5-Dancer-Plugin-DBIC/Makefile
databases/p5-DBIx-Class-InflateColumn-Serializer/Makefile
databases/p5-DBIx-Admin-DSNManager/Makefile
databases/p5-Jifty-DBI/Makefile
databases/p5-Redis/Makefile
databases/py-txredisapi/Makefile
databases/p5-DBIx-Class-InflateColumn-FS/Makefile
databases/p5-BSON-XS/Makefile
databases/py-sqlalchemy14/Makefile
databases/p5-Dancer2-Plugin-DBIC/Makefile
databases/p5-DBIx-Class-AuditLog/Makefile
databases/p5-CHI-Driver-TokyoTyrant/Makefile
databases/py-redis/Makefile
databases/p5-DBIx-Class-EncodedColumn/Makefile
databases/p5-Dancer-Plugin-Database-Core/Makefile
databases/p5-Oryx/Makefile
databases/p5-DBIx-Lite/Makefile
databases/sqlitecpp/Makefile
databases/p5-POE-Component-EasyDBI/Makefile
databases/py-pylibmc/Makefile
databases/p5-DBIx-Simple/Makefile
databases/p5-DBD-MariaDB/Makefile
databases/p5-SQL-Maker/Makefile
databases/py-pymemcache/Makefile
databases/p5-Prophet/Makefile
databases/p5-DBIx-NoSQL/Makefile
databases/py-sqlalchemy13/Makefile
databases/p5-DBIx-Tree/Makefile
databases/p5-DBIx-Skinny-Schema-Loader/Makefile
databases/p5-RedisDB-Parser/Makefile
databases/p5-DBIx-Introspector/Makefile
databases/p5-DBIx-DisconnectAll/Makefile
databases/p5-CouchDB-View/Makefile
databases/py-minidb/Makefile
databases/p5-Class-DBI-Untaint/Makefile
deskutils/py-taskw/Makefile
deskutils/p5-Data-ICal-DateTime/Makefile
deskutils/p5-Data-ICal/Makefile
deskutils/taskwarrior/Makefile
deskutils/py-bugwarrior/Makefile
devel/p5-XS-Parse-Sublike/Makefile
devel/p5-Test-Kwalitee/Makefile
devel/p5-Time-Duration/Makefile
devel/py-botocore/Makefile
devel/p5-self-init/Makefile
devel/p5-Term-Form/Makefile
devel/p5-Data-Section/Makefile
devel/p5-IPC-PerlSSH/Makefile
devel/p5-FFI-Platypus-Type-PtrObject/Makefile
devel/p5-Directory-Scratch-Structured/Makefile
devel/p5-Perl-PrereqScanner-NotQuiteLite/Makefile
devel/pkgconf/Makefile
devel/p5-Test-DependentModules/Makefile
devel/p5-Class-Roles/Makefile
devel/py-jinja2-time/Makefile
devel/R-cran-lintr/Makefile
devel/p5-Test-Base/Makefile
devel/p5-File-Util/Makefile
devel/py-moto/Makefile
devel/R-cran-sessioninfo/Makefile
devel/p5-DateTime-TimeZone/Makefile
devel/p5-Data-Hierarchy/Makefile
devel/p5-Git-Repository/Makefile
devel/p5-Config-Auto/Makefile
devel/p5-File-Write-Rotate/Makefile
devel/p5-Test-WWW-Mechanize-Catalyst/Makefile
devel/py-genson/Makefile
devel/py-rubymarshal/Makefile
devel/p5-Sub-Name/Makefile
devel/p5-Config-Tiny/Makefile
devel/py-omnijson/Makefile
devel/py-pylama/Makefile
devel/p5-Hash-Slice/Makefile
devel/p5-DynaLoader-Functions/Makefile
devel/py-typeguard/Makefile
devel/p5-IO-Prompt-Tiny/Makefile
devel/py-xdoctest/Makefile
devel/p5-Test-Prereq/Makefile
devel/py-columnize/Makefile
devel/trompeloeil/Makefile
devel/p5-MooseX-LazyRequire/Makefile
devel/py-iso8601/Makefile
devel/p5-Data-Serializable/Makefile
devel/py-adb/Makefile
devel/p5-Data-Object-Types/Makefile
devel/p5-Data-Object-Role-Formulatable/Makefile
devel/p5-POE-Quickie/Makefile
devel/p5-github_creator/Makefile
devel/py-rebulk/Makefile
devel/p5-MooseX-Meta-TypeConstraint-Mooish/Makefile
devel/py-jsonschema/Makefile
devel/R-cran-rappdirs/Makefile
devel/p5-IOC/Makefile
devel/py-conditional/Makefile
devel/p5-POEx-Types/Makefile
devel/py-pyperf/Makefile
devel/p5-Proc-Guard/Makefile
devel/p5-Parallel-ForkManager/Makefile
devel/p5-MooseX-Params-Validate/Makefile
devel/p5-Tickit/Makefile
devel/p5-Algorithm-Evolutionary/Makefile
devel/p5-Type-Tiny/Makefile
devel/py-pyhcl/Makefile
devel/py-rauth/Makefile
devel/py-deepdiff/Makefile
devel/py-yappi/Makefile
devel/p5-Log-Dispatch-File-Stamped/Makefile
devel/py-jaraco.functools/Makefile
devel/py-dj40-django-rq/Makefile
devel/py-pudb/Makefile
devel/p5-Data-Validator/Makefile
devel/p5-Test-Mock-LWP-Dispatch/Makefile
devel/py-pytest-subtests/Makefile
devel/p5-Test-File/Makefile
devel/p5-Dist-Metadata/Makefile
devel/p5-Hash-AutoHash-Args/Makefile
devel/p5-Test-Inter/Makefile
devel/py-glob2/Makefile
devel/p5-Data-Object-Role-Dumpable/Makefile
devel/p5-Clone/Makefile
devel/p5-Test-Moose-More/Makefile
devel/jansson/Makefile
devel/p5-Future-IO/Makefile
devel/p5-MooseX-Types-Set-Object/Makefile
devel/rapidcheck/Makefile
devel/p5-Test-MockModule/Makefile
devel/meson/Makefile
devel/py-jdcal/Makefile
devel/p5-MooseX-MetaDescription/Makefile
devel/py-pathspec/Makefile
devel/py-pep8-naming/Makefile
devel/ghq/Makefile
devel/p5-Parse-CPAN-Packages/Makefile
devel/p5-List-SomeUtils-XS/Makefile
devel/p5-BorderStyleBase/Makefile
devel/p5-Module-Build-XSUtil/Makefile
devel/p5-MooseX-Extended/Makefile
devel/p5-MouseX-NativeTraits/Makefile
devel/entt/Makefile
devel/py-fastnumbers/Makefile
devel/p5-File-Flock/Makefile
devel/tcl-trf/Makefile
devel/ros-rosdistro/Makefile
devel/py-extras/Makefile
devel/p5-Sub-Exporter-ForMethods/Makefile
devel/py-line-profiler/Makefile
devel/p5-App-CLI/Makefile
devel/R-cran-ini/Makefile
devel/p5-Devel-MAT/Makefile
devel/p5-Test-DBIC-ExpectedQueries/Makefile
devel/date/Makefile
devel/py-molecule/Makefile
devel/p5-FFI-Platypus-Declare/Makefile
devel/py-buildbot/Makefile
devel/p5-Class-C3-Adopt-NEXT/Makefile
devel/p5-Class-Accessor-Fast-Contained/Makefile
devel/py-stack-data/Makefile
devel/p5-Path-Resource/Makefile
devel/p5-Data-Object-Class/Makefile
devel/py-jupyter-core/Makefile
devel/py-astroid/Makefile
devel/R-cran-mockery/Makefile
devel/p5-Module-Install-ReadmeFromPod/Makefile
devel/p5-Data-Object-Data/Makefile
devel/p5-File-BaseDir/Makefile
devel/py-defusedxml/Makefile
devel/p5-Devel-NYTProf/Makefile
devel/p5-CHI/Makefile
devel/py-Jinja2/Makefile
devel/p5-Test-File-ShareDir/Makefile
devel/py-ubelt/Makefile
^C
dlangille commented 1 year ago

Testing.

[dev-ingress01 dan /jails/freshports/usr/ports] % cd archivers/p5-IO-Compress-Lzf
[dev-ingress01 dan /jails/freshports/usr/ports/archivers/p5-IO-Compress-Lzf] % make -V TEST_DEPENDS
p5-Test-Pod>=1:devel/p5-Test-Pod
% grep TEST_DEPENDS Makefile
TEST_DEPENDS=   p5-Test-Pod>=1:devel/p5-Test-Pod

But, nothing in the database:

freshports.devgit=# select id, element_pathname(element_id), test_depends from ports where id in (select id from ports_active where name = 'p5-IO-Compress-Lzf') order by element_pathname(element_id);
  id   |             element_pathname             | test_depends 
-------+------------------------------------------+--------------
 28244 | /ports/head/archivers/p5-IO-Compress-Lzf | 
(1 row)

freshports.devgit=#
dlangille commented 1 year ago

Let's modify the update script to do just one port:

[dev-ingress01 dan ~/modules/Verify] % svn di
Index: refresh-all-active-ports.pl
===================================================================
--- refresh-all-active-ports.pl (revision 5871)
+++ refresh-all-active-ports.pl (working copy)
@@ -44,6 +44,7 @@
          PA.name,
          PA.pathname as port_pathname
     FROM ports_active PA
+    WHERE name = 'p5-IO-Compress-Lzf'
 ORDER BY category, name;
 ";

[dev-ingress01 dan ~/modules/Verify] %
dlangille commented 1 year ago 
[dev-ingress01 dan ~/modules/Verify] % echo perl ./refresh-all-active-ports.pl | sudo su -fm freshports > /tmp/refresh-p5-IO-Compress-Lzf.log
[dev-ingress01 dan ~/modules/Verify] %
dlangille commented 1 year ago

So that worked....

freshports.devgit=# select id, element_pathname(element_id), test_depends from ports where id in (select id from ports_active where name = 'p5-IO-Compress-Lzf') order by element_pathname(element_id);
  id   |             element_pathname             |           test_depends           
-------+------------------------------------------+----------------------------------
 28244 | /ports/head/archivers/p5-IO-Compress-Lzf | p5-Test-Pod>=1:devel/p5-Test-Pod
(1 row)

freshports.devgit=#
dlangille commented 1 year ago

A revised script is running on dev now. I see progress in the count():

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
    53
(1 row)

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
    95
(1 row)

freshports.devgit=#
dlangille commented 1 year ago

Progress so far:

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
   730
(1 row)
dlangille commented 1 year ago

The grep above, found TEST_DEPENDS in 2332 Makefiles - the script has found 3171 - Perhaps that's from slave ports?

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
  3171
(1 row)
yurivict commented 1 year ago

1242 ports origins have TEST_DEPENDS.

dlangille commented 1 year ago

1242 ports origins have TEST_DEPENDS.

Please show your work. ;)

I want to know how that compares to what I found in https://github.com/FreshPorts/freshports/issues/411#issuecomment-1361498241

yurivict commented 1 year ago
  1. git clone https://github.com/yurivict/freebsd-portsdb.git portsdb
  2. cd portsdb
  3. ./import.sh # this imports /usr/ports into ports.sqlite
  4. sqlite3 ports.sqlite < sql-misc-queries/find-pkgorigins-that-have-test-depends.sql # this finds all pkgorigins with TESTS_DEPENDS
yurivict commented 1 year ago

Sorry, the correct number for pkgorigins with TEST_DEPENDS is 6342. The original query counted them in reverse.

dlangille commented 1 year ago

still going:

freshports.devgit=# select count(*) from ports where test_depends is not null;
 count 
-------
  3823
(1 row)

freshports.devgit=#

FYI, regular commit processing is underway while I do this. That's a first. Sometimes it breaks the script because they are using the same git clone instance. When FreshPorts goes a git pull, sometimes a Makefile is not available. For example:

checking to make sure /jails/freshports/usr/ports/graphics/ocaml-cairo/Makefile exists
 * * * * not found.  WTF?
makecommand = /usr/local/bin/sudo /usr/sbin/jexec freshports /make-port.sh /usr/ports graphics/ocaml-cairo 2>/tmp/FreshPorts.graphics.ocaml-cairo.make-error.2022.12.21.22.35.35.81480
Result = 512
size is 123
getting error message from temp file
$result='-1'
$showconfig=''
That make failed:

'This command (FreshPorts code 1):

/usr/local/bin/sudo /usr/sbin/jexec freshports /make-port.sh /usr/ports graphics/ocaml-cairo 2>/tmp/FreshPorts.graphics.ocaml-cairo.make-error.2022.12.21.22.35.35.81480

produced this error:

Error message is: cd: /usr/ports/graphics/ocaml-cairo: No such file or directory
make: cannot open /usr/ports/graphics/ocaml-cairo/Makefile.
Make results are : 
make: stopped in /
dlangille commented 1 year ago

Looks like the script finished about 9 hours ago:


[dev-ingress01 dan /tmp] % ls -l refresh-all-active-ports.log
-rw-r--r--  1 dan  wheel  609731323 2022.12.22 06:07 refresh-all-active-ports.log
dlangille commented 1 year ago

We have about 5600 ports with TEST_DEPENDS. They are recorded at https://gist.github.com/dlangille/c797354c61ac2f9778fc3c0010243f50

dlangille commented 1 year ago

Taking one at random, I see: https://dev.freshports.org/devel/py-inifile/

dlangille commented 1 year ago

NOTES for me: when updating a database:


begin;
drop view ports_all;
drop view ports_active;
\i updates-2022-12-19-test-depends.ddl 
\i views.sql 
\i permissions.sql 
commit;
dlangille commented 1 year ago

After more jail configuration this morning, the update script is running:

[r720-02-ingress01 dan /usr/local/etc/sudoers.d] % tail -F /tmp/refresh-all-active-ports.log | grep skipp
skipping update - no change in test_depends
skipping update - no change in test_depends
skipping update - no change in test_depends
skipping update - no change in test_depends
dlangille commented 1 year ago

Guess who didn't start this in tmux... restarting.

dlangille commented 1 year ago

This isn't adding up. It's as if the script does not complete.

freshports.org=# select count(*) from ports where test_depends is not null;
 count 
-------
   155
(1 row)
[r720-02-ingress01 dan /jails/freshports/usr/ports] % grep -lr TEST_DEPENDS * | wc -l]\
    2332
dlangille commented 1 year ago

It didn't get past audio/rezound

[r720-02-ingress01 dan /jails/freshports/usr/ports] % tail /tmp/refresh-all-active-ports.log
RepoType='git'
Repository='head'
CommitBranch='0'
calling FreshPorts::Branches::GetPathToRepoForBranch
CommitBranch:   '0
REPODIR:        '/usr/ports'
REPODIR_CHROOT: '/usr/ports'
checking to make sure /jails/freshports/usr/ports/audio/rezound/Makefile exists
Phew.  It's here.  Moving on....
makecommand = /usr/local/bin/sudo /usr/sbin/jexec freshports /make-port.sh /usr/ports audio/rezound 2>/tmp/FreshPorts.audio.rezound.make-error.2022.12.22.19.5.43.25716
[r720-02-ingress01 dan /jails/freshports/usr/ports] %
dlangille commented 1 year ago

Still running, looking better:

freshports.org=# select count(*) from ports where test_depends is not null;
 count 
-------
  3143
dlangille commented 1 year ago

This started at 0208 and it's now 1957:

[r720-02-ingress01 dan ~] % ps auwwx
USER         PID %CPU %MEM    VSZ    RSS TT  STAT STARTED     TIME COMMAND
freshports 89298  2.2  0.0 174740 121384  2  S+J  02:18   11:16.88 perl refresh-all-active-ports.pl
dlangille commented 1 year ago

the backend and frontend code for this are both in production. The test_depends field is not yet populated. New commits should populate it if there is a value specified.

dlangille commented 1 year ago

Done.


freshports.org=# select count(*) from ports where test_depends is not null;
 count 
-------
  6329
(1 row)

freshports.org=#

Over to you. :) Merry Christmas.

dlangille commented 1 year ago

Full list, with test_depends values, at https://gist.github.com/dlangille/ef20e4729ac89231d7df32a5ca0968e7

dlangille commented 1 year ago

However, https://r720-02.freshports.org/x11/xpra/ does not show any test_depends

https://dev.freshports.org/astro/py-astral/#dependencies does show it.

Perhaps your code needs a pull from the repo for classes/port-display - https://github.com/FreshPorts/freshports/commit/4b90a283c22575debe2c30ba5220cd9e6dd0af26

dlangille commented 1 year ago

@yurivict your test instance now processes live commits. It should always be up to date with prod.

dlangille commented 1 year ago

@yurivict Hope all is well. :)