the new category broke fp-listen

[dlangille]

The new category, budgie, broke the front end cache clearing: i.e. fp-listen

The commit: https://cgit.freebsd.org/ports/commit/?id=1ccad74cc8c19d8a0fa0ab573604db5c3484d90d

The logs:

Mar  4 18:22:08 dev-nginx01 fp-listen[69366]: We have a new category
Mar  4 18:22:08 dev-nginx01 fp_listen[69365]: Traceback (most recent call last):
Mar  4 18:22:08 dev-nginx01 fp_listen[69365]:   File "/usr/local/lib/python3.9/site-packages/fp-listen/fp-listen.py", line 298, in <module>
Mar  4 18:22:08 dev-nginx01 fp_listen[69365]:     ProcessCategoryNew()
Mar  4 18:22:08 dev-nginx01 fp_listen[69365]:   File "/usr/local/lib/python3.9/site-packages/fp-listen/fp-listen.py", line 206, in ProcessCategoryNew
Mar  4 18:22:08 dev-nginx01 fp_listen[69365]:     urllib.urlretrieve("http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/www/en/ports/categories?rev=HEAD;content-type=text%2Fplain", '/usr/websites/freshports.org/dynamic/caching/tmp/categories');
Mar  4 18:22:08 dev-nginx01 fp_listen[69365]: AttributeError: module 'urllib' has no attribute 'urlretrieve'

Seems we haven't had a new category since we went to git.

http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/www/en/ports/categories?rev=HEAD;content-type=text%2Fplain

will go to:

https://svnweb.freebsd.org/ports/head/en/ports/categories

We have a way to get a list of categories. That is done at the start of processing for each commit and is printed at the top of the log file. Example:

[dev-ingress01 dan ~freshports/message-queues/recent] % head 2023.03.05.00.46.13.000002.d37deb7950903f81accc32c91328f39a750b2c6d.log\
grabbing categories from disk
'accessibility arabic archivers astro audio benchmarks biology cad chinese comms converters databases deskutils devel distfiles dns editors emulators finance french ftp games german graphics hebrew hungarian irc japanese java korean lang mail math misc multimedia net net-im net-mgmt net-p2p news polish ports-mgmt portuguese print russian science security shells sysutils textproc ukrainian vietnamese www x11 x11-clocks x11-drivers x11-fm x11-fonts x11-servers x11-themes x11-toolkits x11-wm'
categories already loaded
about to process
finished setting up the Parser
Processing file [/var/db/ingress/message-queues/incoming/2023.03.05.00.46.13.000002.d37deb7950903f81accc32c91328f39a750b2c6d.xml]...
parsing file now

 *** start of all updates ***
[dev-ingress01 dan ~freshports/message-queues/recent] % 

But that does not include virtual categories, which budgie is.

[dlangille]

It's good that https://dev.freshports.org/budgie/ loads and looks as expected.

First task: find out what fp-listen wants with that list. It may no longer be relevant.

[dlangille]

For those coming from social media: this was fixed with sudo service fp_listen start

The symptoms: no commits after the crash appeared on the website. They were correctly processed by the backend (aka ingress). With the next commit after the service was restored, everything on the website was up to date. I could have cleared the cache after starting the service, but I'm babysitting this weekend.

[dlangille]

TO DO: use the -r flag: re https://twitter.com/vermaden/status/1636504683052167171

EDIT: 2023-10-17 - it has been done.

[dlangille]

First task: find out what fp-listen wants with that list. It may no longer be relevant.

It sets flags:

  Touch(config['flags']['WWWENPortsCategoriesFlag'])
  Touch(config['flags']['JOBWAITING'])

Which will be processed by job-waiting.pl, which will invoke process_www_en_ports_categories.sh, which invokes categories_update_descriptions.pl, passing in ${SPOOLINGDIR}/categories which is a copy of the downloaded file from http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/www/en/ports/categories?rev=HEAD;content-type=text%2Fplain

Sounds like we need a new way.

[dlangille]

make -V VALID_CATEGORIES gives a list of all categories, virtual and physical.

Combined with modules/categories.pm which can get a list of categories on disk, we can determine virtual vs physical.

Descriptions for physical categories are in their respective Makefile; for example:

[19:17 dev-ingress01 dan /jails/freshports/usr/ports/accessibility] % grep COMMENT Makefile 
    COMMENT = Ports to help disabled users

For virtual categories, you can look in Mk/Scripts/desktop-categories.sh:

[19:28 dev-ingress01 dan /jails/freshports/usr/ports] % grep -r budgie Mk/Scripts/*
Mk/Scripts/desktop-categories.sh:       budgie)         c="Budgie GTK"                          ;;

One might be able to script a list out of that file.