lcn2 commented 4 months ago

TODO

Initial TODO list

The following items need to be done before this issue may be considered complete. See below for details and terms.

[x] While issue #3 is open, update the IOCCC manifest as per new/renamed/deleted files in the IOCCC YYYY/dir entry tree
[x] Once issue #3 is closed, verify that the IOCCC manifest matches the IOCCC YYYY/dir entry tree
[ ] Add download property to files that must be downloaded
[ ] Verify that certain manifest properties are reasonably consistent within the IOCCC manifest.

See also Comment 2054436075.

There may need to be exceptions, but for the most part similar rows in the manifest.numbers spreadsheet should be reasonably consistent in terms of these fields:

This TODO item applies to the following manifest properties:

inventory_order
OK_to_edit
display_as
display_via_github
entry_text

It may be OK for a manifest property exception to exist if there is a good reason to do so.

Pay particular attention to the _entrytext field. Be sure it is a short but useful description string of no more than 59 ASCII characters and does not contain what looks like an HTML element.

[ ] Verify that issue #3 is complete

:-)

[ ] Perform one final inspection of the IOCCC manifest

Pay attention to the manifest properties that should be used consistently across the IOCCC manifest.

Final TODO list

List the consistency exceptions to the manifest properties.

[ ] Generate all YYYY/dir entry .entry.json files based on the final IOCCC manifest
[ ] Generate all YYYY/dir entry index.html files based on the final IOCCC manifest
[ ] Declare the manifest in YYYY/dir entry .entry.json files to be truth for the IOCCC manifest

Move the manifest numbers spreadsheet, and related files and scripts in tmp under the tmp/OLD_manifest directory.

What is the IOCCC manifest?

The IOCCC manifest is complete list of files for all IOCCC entries. The IOCCC manifest for each winning entry is described in the manifest JSON array in the JSON file .entry.json located in each IOCCC YYYY/dir entry directory.

HOWEVER: As of 2024 Jan 18, the source of truth for the IOCCC manifest the temporary file manifest numbers spreadsheet and temporary set of tools that, in turn produces a set of files that in turn generates a .entry.json file located in each IOCCC YYYY/dir entry directory. Until the tool that will generate the .entry.json files is created, we must maintain the manifest numbers spreadsheet and use the temporary set of tools to produce those .entry.json files located in each IOCCC YYYY/dir entry directory.

Q: When will the source of truth migrate to the .entry.json files located in each IOCCC YYYY/dir entry directory?

A: When this issue is completed. The goal is this issue is to use the improved IOCCC manifest to generate a worthy set of .entry.json files located in each IOCCC YYYY/dir entry directory.

Until this issue is resolved, consider the temporary file manifest numbers spreadsheet as the IOCCC manifest.

Caution: numbers spreadsheets look like a single blob to git

The manifest numbers spreadsheet appears as a binary blob to git(1). One cannot do a useful git diff on a number spreadsheet. The consequence of this is that it is easy to lose previous edits if one is not careful. Simply doing an edit and then undoing that edit will change the number spreadsheet such that a git commit will overwrite an edit from a previous commit.

Before doing a git commit that involves the manifest numbers spreadsheet, please make an explicit check modification date of the manifest numbers spreadsheet:

git archive HEAD tmp/manifest.numbers | tar -tvf - tmp/manifest.numbers

If needed, explicitly fetch the current manifest numbers spreadsheet:

git archive HEAD tmp/manifest.numbers | tar -xvf - tmp/manifest.numbers

and apply your manifest changes to that updated file.

See Find the Difference Between Two Tables In Mac Numbers for an idea of how to compare two tables.

How to verify that numbers manifest matches the entry file tree

The following procedure will verify that the numbers manifest matches the entry file tree:

Step -1

Verify that you have the current manifest numbers spreadsheet file:

git archive HEAD tmp/manifest.numbers | tar -tvf - tmp/manifest.numbers

If needed, explicitly fetch the current manifest numbers spreadsheet:

git archive HEAD tmp/manifest.numbers | tar -xvf - tmp/manifest.numbers

Step 0

Verify that your YYYY/dir entry file tree is clean:

git status

If the status does not report nothing to commit, working tree clean, remove, or stash away, or commit changes as needed.

1) Verify the repo is up to date

git fetch && git rebase

Step 2

Generate a new manifest CSV file:

Step 2a

Move into the repo tmp directory as things are easier to perform form within that directory.

cd tmp

Step 2b

Open the manifest numbers spreadsheet in macOS numbers:

open manifest.numbers

Step 2c

Export the CSV file:

In numbers: File -> Export To -> CSV ...

Export the manifest.csv into the tmp directory of the repo, overriding the existing manifest.csv file.

Step 2d

Normalize the manifest.csv file and if needed, update the tmp/path_list.required.txt, tmp/c, and tmp/path_list.manifest.txt files:

./fix_csv.sh

One aspect of the `tmp/fix_csv.sh tool is to normalize the CSV file by terminating all lines with a newline, and to end the file with a final newline.

Another aspect of the `tmp/fix_csv.sh tool is to generate:

path_list.required.txt - list of files that are required to exist
path_list.required.txt - list of files that are constructed and are OK to not exist

BTW: The tmp/fix_csv.sh tool will also normalize the other CSV files: tmp/author.csv, tmp/author_wins.csv, and tmp/year_prize.csv as well.

Another job of the tmp/fix_csv.sh tool is to detect the following types of errors:

cells with leading whitespace
cells with trailing whitespace
duplicate year,dir,file_path entries

I.e., a file is found more than once in the manifest.

cells with "TRUE" instead of "true"

JSON requires "true" while Numbers, by default will convert to "TRUE" unless the cell is formatted as text.

cells with "FALSE" instead of "false"

JSON requires "false" while Numbers, by default will convert to "FALSE" unless the cell is formatted as text.

cells with "null"

The manifest does not use the JSON null value, so it is an invalid manifest cell value.

cells with "NULL"

JSON requires "null" while Numbers, by default will convert to "NULL" unless the cell is formatted as text. However the manifest does not use the JSON null value, so it is an invalid manifest cell value.

empty cells

The manifest does not allow for a cell value to be empty.

HINT: By default, the conditional formatting of the manifest numbers spreadsheet will turn the background of an empty cell to orange.

Step 3

Generate the list of YYYY/dir entry files by running the tool:

./gen_path_list.found.sh

The file tmp/path_list.found.txt will contain a complete list of YYYY/dir entry files.

NOTE: This tool does a make clobber (which is why the tool seems to pause) before generating the list of YYYY/dir entry files.

HINT: You can get a preview of any changes made, since the last update of tmp/path_list.found.txt by doing:

git diff path_list.found.txt

If there are differences in tmp/path_list.found.txt, chances are good that the manifest numbers spreadsheet needs to be updated accordingly.

Step 4

Verify that the manifest file list matches the current set of required YYYY/dir entry files:

Run the following tool to compare the manifest numbers spreadsheet with the required YYYY/dir entry files:

./check_path_list.sh

Here is an example of an tmp/check_path_list.sh error message (we fake some of these messages):

# ./check_path_list.sh: Warning: missing required file count: 1
#
# ./check_path_list.sh: missing required file list starts below
1984/anonymous/Makefile
# ./check_path_list.sh: missing required file list ends above
# ./check_path_list.sh: Warning: found files not in the manifest count: 5
#
# ./check_path_list.sh: found files not in the manifest list starts below
1995/heathbar/try.sh
1995/savastio/try.sh
1995/schnitzi/try.sh
1995/vanschnitz/try.sh
1996/dalbec/try.sh
# ./check_path_list.sh: found files not in the manifest list ends above
./check_path_list.sh: Warning: about to exit: 19

HINT: A similar error list can be generated via the command:

git diff path_list.found.txt

IMPORTANT: If the tmp/check_path_list.sh prints an error message and/or edits non-zero, do not proceed to step 5! Instead, you must update the manifest numbers spreadsheet to account for the reported changes to the YYYY/dir entry files. Go back to Step -1 🙁, update the manifest numbers spreadsheet accordingly, and then proceed to Step 0 and complete this process.

HINT: Use the ability to sort columns find how similar files for other YYYY/dir entries are listed. For example, in the above error message from tmp/check_path_list.sh, we need to add severaltry.shentries. Sort by _file_path_, scroll down to the bottom of thetry.shentries, open up new rows below it (in this case we select the bottom 5try.shrows, right-click and select _Add Rows Below_ and fill in the empty (by default colored orange) cells). By copying and pasting columns C thru H, we can make the newtry.sh` consistent with the previous values. However, don't forget to canonically resort the spreadsheet (see below).

REQUEST: Keep the manifest numbers spreadsheet canonically sorted. See below for how to canonically sort the spreadsheet.

REQUEST: When creating new entry files, renaming existing entry files, or removing entry files, please remember update the manifest! Use the steps in this procedure to verify that the manifest has been appropriately updated.

If the tmp/check_path_list.sh prints nothing and exits 0, all is well. You may proceed to step 5.

Step 5

Verify that the manifest numbers spreadsheet has not changed out from under you:

git archive HEAD manifest.numbers | tar -tvf - manifest.numbers

If it was updated while you were working you need to BACK to step -1, and start again. 😖

Step 6

Check everything in:

git add -u
git status
git commit -m'update manifest as per recent file changes'
git push

At this point one must hope that between steps 5 and 6 the manifest numbers spreadsheet was not updated.

At the end of this step, the manifest is now in sync with the files in all IOCCC YYYY/dir entry directories.

Update `.entry.json` files

Step 7

Generate updated .entry.json files:

Assuming the tmp/check_path_list.sh did not print any error messages and exited 0, rebuild a complete set of entry.json files:

./run_all.sh ./gen_entry_json.sh

NOTE: The above command will take a few seconds to run on most systems.

HINT: To generate a.entry.json for a single IOCCC YYYY/dir entry, run the tmp/gen_entry_json.sh directly. For example, with a little verbosity:

./gen_entry_json.sh -v 3 2020/ferguson1

Step 8

Examine the differences:

git diff

If all is well, proceed to Step 9, otherwise go back to Step -1 😕 and correct.

At the end of this step, all copies of the IOCCC manifest (both the manifest numbers spreadsheet and the complete set of all IOCCC entry.json files) are now in sync with the files in all IOCCC YYYY/dir entry directories. That is, the IOCCC manifest is up to date.

Step 9

Check everything in:

git add -u
git status
git commit -m'update .entry.json files per recent manifest changes'
git push
cd .. # cd to the top of the repo

Update entry `index.html` files

Step 10

Update all entry index.html files:

bin/all-run.sh -v 3 bin/quick-readme2index.sh -v 1

NOTE: The quick-readme2index.sh tool assumes that if index.html exists as a non-empty files PLUS if index.html is newer than both .entry.json and README.md then the tool will quickly exit, otherwise the tool runs the readme2index.sh to r build the index.html file.

To be sure that all entry index.html files are correctly built, run this command that forces ALL index.html files to be rebuilt:

bin/all-run.sh -v 3 bin/readme2index.sh -v 1

While certain, the above command takes longer, but Sorry, (tm Canada 🇨🇦), this form of the comment take a while to complete.

So if one makes a few changes to the manifest, and sometime in the past the above "readme2index.sh" ran to the end, using the previously mentioned "quick-readme2index.sh" command is a faster way to complete this step.

Step 11

Examine the differences:

git diff

If all is well, proceed to Step 12, otherwise go back to Step -1 😣 and correct.

Step 12

Check everything in:

git add -u
git status
git commit -m'update index.html and .entry.json per recent manifest changes'
git push

At the end of this step, all entry index.html files are in sync with the IOCCC manifest.

To canonically sort the manifest numbers spreadsheet

With the manifest numbers spreadsheet open in macOS numbers, click on the cell A2 (the top non-header cell). Then select Organize (top row button close to the right-hand edge: you may have to click the ">>" to see it) -> Sort -> click ((Sort Now))

The canonical sort is by:

a) year - ascending b) dir - ascending c) inventory_order - ascending d) file_path - ascending

Manifest fields

For each entry file in the manifest, there exists 8 properties.

In the case of the manifest numbers spreadsheet, this is the column header of the spreadsheet.

In the case of the entry.json file, this is the JSON member name of the element of the manifest JSON array.

What follows are the exists 8 properties, listed in spreadsheet column order, and JSON member name order:

year

Purpose: The IOCCC year of the win.

Normally 'year' is a 4-digit number, however mock is also possible.

dir

Purpose: The directory under the 'year'.

The _entryid consists of the year followed by an underscore ("_") followed by the dir.

file_path

Purpose: A file in the entry directory.

Unless the YYYY/dir entry has sub-directories, then file_path is just the basename. However, if the has sub-directories, then file_path will contain a slash ("?"). E.g., img/fs.tar.

inventory_order

Purpose: The order in which this member is listed in the YYYY/dir entry's inventory near the bottom of the entry index.html file.

The inventory_order must be an integer >= 0.

The lower the inventory_order, the higher up the given file is listed in the inventory near the bottom of the IOCCC YYYY/dir entry index.html file.

If inventory_order <= 999999999, the file is considered a primary file. All primary files are listed in the top section of the inventory near the bottom of the IOCCC YYYY/dir entry index.html file.

If inventory_order >= 1000000000, the file is considered a secondary file. All secondary files are listed in the bottom section of the inventory near the bottom of the IOCCC YYYY/dir entry index.html file. By convention, we use _4000000000_for most secondary files, although any integer >= 1000000000 will work.

For compressed tarball files (i.e., files that end in .tar.bz2 or files with a _displayas value of tbz2) we recommend a inventory_order value of 1415926535 so that such files will usually appear at the top of of the secondary files section.

For entry index.html files we recommend a inventory_order value of 4294967295 so that index.html files will usually appear at the end of secondary_ files section.

If two files in an entry have the the same _inventoryorder then the file_path with the lowest dictionary order is listed first.

NOTE: If is a very good idea if the use of inventory_order was reasonably consistent across the entire IOCCC inventory.

NOTE: If would be a good idea if similar types of files had a similar inventory_order. For example, the main C source for each IOCCC YYYY/dir entry might have the same low inventory_order so that the entry appears high up in the inventory. The Makefile have a slightly larger inventory_order. Alt C source might have an even larger inventory_order. The original C source might have an even larder inventory_order. The suggestion is to try, where possible, to be consistent across all winning IOCCC entities.

OK_to_edit

Purpose: A JSON boolean indicating if it is OK to edit this file.

An OK_to_edit of true means that the file is NOT a file that is built by some program or bin directory tool. That is, a true OK_to_edit is some file that someone might choose to modify or remove as part of a pull request.

An OK_to_edit of false means that the file is a file that built by some program, OR it is an original source file (i.e., "prog.orig.c"). That is, a false OK_to_edit file should NOT be modified nor removed part of a pull request.

Examples of false_ OK_to_edit file are:

index.html

These files are constructed by the bin/readme2index.sh tool.

prog.orig.c

These files are the original winning YYYY/dir entry C source that own the IOCCC.

yyyy_dir.tar.bz2

These files are generated by the bin/tar-entry.sh tool.

.path

These files are generated by the make genpath from the top or year level Makefile.

NOTE: If is a very good idea of the use of OK_to_edit was reasonably consistent across the entire IOCCC inventory.

display_as

Purpose: The type of file to be displayed.

While the display_as is not currently used, in the future this value may guide how a given file is displayed. For example, a display_as of c would indicate that the file should be displayed as a C source file with possible color values that correspond to the type of C element: along the lines of how vim(1) colors certain parts of C source.

NOTE: The display_as is NOT a file(1) utility output as that tends to be much more detailed that is needed. The display_as is more of a way to group similar file together in order to help guide a future in-browser display process.

NOTE: If is a very good idea if the use of display_as was reasonably consistent across the entire IOCCC inventory.

display_via_github

Purpose: A JSON boolean indicating if the file's inventory link should be displayed from the GitHub repo.

A display_via_github of true means that the file's inventory link should directly reference the GitHub repo file.

A display_via_github of false means that the file's inventory link should reference the web site.

The display_via_github of true means that when one clicks on the file in the inventory, one will jump to the GitHub repo file as indicated by the master branch. This allows the use of the way GitHub displays files, such as C source and shells script source. One might wish to use a true display_via_github for C source, shell scripts, Makefiles, .gitignore files, and JSON files.

If one wishes to display the contents of a file, and the browser would otherwise simply download the file, then a display_via_github of true should be used.

If displaying a file with color is helpful, as with C, shell scripts, and JSON. then display_via_github of true should be used.

The display_via_github of false means that the browser will attempt to visit the file from the web site. This might cause the file to be rendered and displayed in the browser (such as a picture image file), or it might cause the file to be downloaded.

If one wishes to cause a file to be dowloaded (such as the compressed tarball or some binary data file), and the browser is willing to download the file (instead of displaying it), then a display_via_github of false should be used.

If a browser can display the contents of a file (such as picture image), then a display_via_github of false should be used.

NOTE: If is a very good idea if the use of display_via_github was reasonably consistent across for common files the entire IOCCC inventory. Less common files, such as data files particular to a given YYYY/dir entry, do not need to be as consistent.

entry_text

Purpose: When the file is listed in the listed in the YYYY/dir entry's inventory near the bottom of the entry index.html file, after the file link there will be a dash ("-") followed by the entry_text string.

IMPORTANT: The entry_text should be a very short string, say no longer than 59 characters. This is to keep the inventory liens short. Moreover, the entry_text should NOT contain any non-ASCII characters, no characters that would look like it was part of an HTML element.

NOTE: If is a very good idea if the use of entry_text was reasonably consistent across the entire IOCCC inventory.

UPDATE 0

We have updated this comment as per changes made by aedb0172239504fb14249a23069e91367b10b58b. We may have missed a changes in this TODO comment.

xexyl commented 4 months ago

I hope this is okay to say just this even though you say not to reply (in fairness this is not replying to the content): please assign this to me too. Thanks. I'll hold off otherwise until later when it's ready.

lcn2 commented 4 months ago

I hope this is okay to say just this even though you say not to reply (in fairness this is not replying to the content): please assign this to me too. Thanks. I'll hold off otherwise until later when it's ready.

Ready for action.

xexyl commented 4 months ago

I hope this is okay to say just this even though you say not to reply (in fairness this is not replying to the content): please assign this to me too. Thanks. I'll hold off otherwise until later when it's ready.

Ready for action.

But I should still focus on #3 but then if I see something in an entry I can just update it here or should I just make a note? Thanks.

lcn2 commented 4 months ago

I hope this is okay to say just this even though you say not to reply (in fairness this is not replying to the content): please assign this to me too. Thanks. I'll hold off otherwise until later when it's ready.

Ready for action.

But I should still focus on #3 but then if I see something in an entry I can just update it here or should I just make a note? Thanks.

focus on #3 but then if I see something in an entry I can just update it here

xexyl commented 4 months ago

I hope this is okay to say just this even though you say not to reply (in fairness this is not replying to the content): please assign this to me too. Thanks. I'll hold off otherwise until later when it's ready.

Ready for action.

But I should still focus on #3 but then if I see something in an entry I can just update it here or should I just make a note? Thanks.

focus on #3 but then if I see something in an entry I can just update it here

I guess you mean I should do what I suggested? If that's the case I'll have to review the details in this issue more thoroughly but that probably won't be today: perhaps tomorrow. But even so I will be going through it eventually and it won't take more time to do it that way anyway.

Anyway if I see something I'll fix it. I am aware that there is a manifest issue in one of my entries - an incorrect file name (from another entry .. not sure if it's in that other entry). Well one of the files does refer to a file in another entry. I know it's in the Numbers spreadsheet too which is what happened I guess (copy paste error?).

lcn2 commented 4 months ago

BTW: The core of the functionality of bin/readme2html.md is about to be moved into a new command. It doesn't impact the above steps, however.

xexyl commented 4 months ago

BTW: The core of the functionality of bin/readme2html.md is about to be moved into a new command. It doesn't impact the above steps, however.

Thanks for the warning.

xexyl commented 4 months ago

I will get back to 3 in a while. I hope. I don't know how much I can do today but I hope to get 2006 done.

Even so I am not sure about the previous years wrt the manifest. This is partly because I don't know how updated it is so I don't know how far it goes back.

If you could update it up through the website one entry that I did in 2006 I can continue from there.

ALTERNATIVELY

If that's a problem (including inconvenient wrt what you're trying to work on now!) maybe you could tell me the last entry year that was updated? Then I can just go from there.

xexyl commented 4 months ago

I just took a look at the OP. It seems like there might be a procedure to somewhat automate the process of getting the files.

One problem is that I have a lot of other files lying about: for different reasons.

So if I am correct (please let me know) that that procedure above will do what is needed then I will clone in another directory my fork and then I will have a clean state. I can then pull as necessary and update after that.

It might mean that I will have to do it the next day but that might be better anyway because it happens often enough that I might not be able to do the procedure at the end of the day especially as sometimes I return to it to make a quick change.

Anyway please advise. I have to stop typing with the phone: my ulnar nerve is quite sore.

Will be at laptop later on.

lcn2 commented 4 months ago

I will get back to 3 in a while. I hope. I don't know how much I can do today but I hope to get 2006 done.

Even so I am not sure about the previous years wrt the manifest. This is partly because I don't know how updated it is so I don't know how far it goes back.

If you could update it up through the website one entry that I did in 2006 I can continue from there.

ALTERNATIVELY

If that's a problem (including inconvenient wrt what you're trying to work on now!) maybe you could tell me the last entry year that was updated? Then I can just go from there.

The manifest of the past is irrelevant. What matters is what the winner is NOW.

This is why we have Step 4: to be sure that the IOCCC manifest accurately reflects the winner directory *NOW**.

What is very important, and is the real aim of this issue beyond a simple accounting of the winner directories *NOW**, are the other manifest field values for a given entry as they have impact on how a winners inventory is presented in the winner's index.html.

The _inventoryorder, _OK_toedit, _displayas, _display_viagithub, and especially _winnerstext need checking and improving. Their values were hastily put into the manifest numbers spreadsheet just to get values into those fields. The manifest was put into spreadsheet form so that similar types of files (e.g., try scripts), could be somewhat consistent across the IOCCC web site.

The real work of this issue, beyond the simple act of accounting for all files in a winner directory, is to improve those other field values in terms of both consistency across the repo, in terms of accuracy (say, if the link should go to GitHub or not), and usefulness (in terns of order, an especially the short _winnerstext string), etc.

xexyl commented 4 months ago

I will get back to 3 in a while. I hope. I don't know how much I can do today but I hope to get 2006 done. Even so I am not sure about the previous years wrt the manifest. This is partly because I don't know how updated it is so I don't know how far it goes back. If you could update it up through the website one entry that I did in 2006 I can continue from there.

ALTERNATIVELY

If that's a problem (including inconvenient wrt what you're trying to work on now!) maybe you could tell me the last entry year that was updated? Then I can just go from there.

The manifest of the past is irrelevant. What matters is what the winner is NOW.

Okay. This is why we have Step 4: to be sure that the IOCCC manifest accurately reflects the winner directory NOW*.

Okay.

What is very important, and is the real aim of this issue beyond a simple accounting of the winner directories NOW*, are the other manifest field values for a given entry as they have impact on how a winners inventory is presented in the winner's index.html.

Which I guess you mean the values you list below?

The _inventoryorder, _OK_toedit, _displayas, _display_viagithub, and especially _winnerstext need checking and improving. Their values were hastily put into the manifest numbers spreadsheet just to get values into those fields. The manifest was put into spreadsheet form so that similar types of files (e.g., try scripts), could be somewhat consistent across the IOCCC web site.

Consistency is good of course.

The real work of this issue, beyond the simple act of accounting for all files in a winner directory, is to improve those other field values in terms of both consistency across the repo, in terms of accuracy (say, if the link should go to GitHub or not), and usefulness (in terns of order, an especially the short _winnerstext string), etc.

So the obvious question is what was your intention with those fields? I can guess:

inventory_order specifies the order to show the files.
OK_to_edit says whether the file can be edited (example where it can't be is the .orig.c files).
display_as I am not sure about as far as possible values go.
display_via_github would I guess indicate whether to download or to show on GitHub. But if it's on the ioccc.org website what does it mean then?
winners_text I am not sure about either.

Understand that I am so exhausted today that perhaps if I could look at examples it might be clear but at least one above has a question to think about. It seems like I might not be able to do much here (or any other issue) today. I'm sorry. I don't know why I'm so exhausted. I hope that changes later on but it seems not very likely at this point.

UPDATE 0

Okay I see that the fields are described above. I'll see if I can parse them right now. If not it'll come another time possibly not until another day (presumably tomorrow).

xexyl commented 4 months ago

Going through this but I guess the fact I have files not kept in git control is going to be a problem and I will have to do another clone ... will find out in a moment.

xexyl commented 4 months ago

.. yup .. it did. I have to do another clone first.

Of course it's taking a long time ... so have to hold on. And I might be going afk soon. But I believe that I can take care of this today. Even so I'm only doing 2006. I don't know when you last updated it. Perhaps you could let me know so I can take care of that too?

xexyl commented 4 months ago

Oh good. You took care of the checking for files that exist that are not in the manifest. Thanks! That will help a lot. I do have several files to add but I will be going afk soon and so I will have to delay finishing this until after I'm back.

xexyl commented 4 months ago

Manifest updated .. doing next steps now!

UPDATE 0

./run_all.sh ./gen_winner_json.sh

sure takes a while! Next time I'll do it all in one pull request though. Well I might anyway but I'll keep it as different commits as you requested.

xexyl commented 4 months ago

Running bin/all-run.sh -v 3 bin/readme2index.sh -v 1 now .. if all good I'll do a commit and then push for a single pull request (with multiple commits of course). Not sure if I'll manage anything else today. It might be nice if a script could automate all the scripts but the problem I guess is it would maybe end up with a single commit. Another problem is I don't feel up to writing such a script and it might even be more error prone anyway (probably is but it would be convenient).

Anyway hopefully soon it'll be ready.

UPDATE 0

Seems good I ran the longer one too .. I hope. It seems some index.html files were out of date.

UPDATE 1

Ah .. the out of date index.html files is due to the change to ZZ from null (or whatever it was before).

But only at 2000 so will be a while.

xexyl commented 4 months ago

To not slow things down so much I have decided that I will not update the manifest every time there are file changes. Instead I will do it every three or four days (every week might be easier to manage) so that this way I can get more done. Yesterday proved to me that focusing on the manifest slows me down a fair bit. True yesterday I was exceptionally tired (today seems a bit better at least!) but even so it did slow me down.

I am going to set a reminder for every Saturday. If you wish me to do it twice a week I can do it say every Saturday and every Wednesday. I will set the reminder to 0400 so t hat I am sure to see it earlier on which means it'll be one of the first things I do at the repo. I believe this is a better way to handle it.

I am also contemplating writing a script that automates some of the above. It would check the exit codes, run git diff, telling them to look at it very carefully and then after git diff exits it will prompt them if everything is okay. If they input anything but y or Y it will tell them to start over and then try again. It would also first ask them (before doing anything .. after parsing args I guess but maybe even before to save some time) if they are sure the manifest files are up to date.

Now about the manifest I had a thought. Wouldn't it be easier to just have the CSV file? Updating the spreadsheet takes more time and effort. Also I have to say the text is rather small for my vision: it was quite hard to see. Now as for the vision problem an alternative is to increase the font size. Do you mind if I do this? It would make it much easier for me. But if the spreadsheet is not strictly necessary and updating the CSV file is sufficient that would be hugely helpful too. Either way if you feel the spreadsheet IS necessary I will do it that way but increasing the font size would be EXTREMELY helpful. I would have seen (maybe?) the mistake made yesterday earlier on (but maybe not .. who knows). What do you think about these things?

lcn2 commented 4 months ago

I am going to set a reminder for every Saturday. If you wish me to do it twice a week I can do it say every Saturday and every Wednesday. I will set the reminder to 0400 so t hat I am sure to see it earlier on which means it'll be one of the first things I do at the repo. I believe this is a better way to handle it.

Twice a week should be OK. From experience, once you do it a few times it should become rather easy and manageable.

UPDATE 0

As you mostly are adding try shell scripts, you can sort by the filename column, open up enough lines for the new set of try scripts, copy and paste the other commands n columns, copy and paste the years you worked on, add in the winner directly names (using autocompletion helps), then resort the whole spreadsheet in canonical order before saving.

lcn2 commented 4 months ago

I am also contemplating writing a script that automates some of the above. It would check the exit codes, run git diff, telling them to look at it very carefully and then after git diff exits it will prompt them if everything is okay. If they input anything but y or Y it will tell them to start over and then try again. It would also first ask them (before doing anything .. after parsing args I guess but maybe even before to save some time) if they are sure the manifest files are up to date.

If you feel the need to do this, please put such tool under tmp/. This process is temporary, until this issue #1933 is resolved.

lcn2 commented 4 months ago

Now about the manifest I had a thought. Wouldn't it be easier to just have the CSV file? Updating the spreadsheet takes more time and effort.

Sorry (tm Canada 🇨🇦) but until issue #1933 is resolved, we need the numbers spreadsheet to look across the entire manifest in various ways.

lcn2 commented 4 months ago

Also I have to say the text is rather small for my vision: it was quite hard to see. Now as for the vision problem an alternative is to increase the font size. Do you mind if I do this? It would make it much easier for me.

By all means, change the font size if that helps you see the data better.

UPDATE 0

You will need to have a usable spreadsheet to complete issue #1933, especially when it comes time to make certain types of decisions about consistency across all winner directories.

xexyl commented 4 months ago

Now about the manifest I had a thought. Wouldn't it be easier to just have the CSV file? Updating the spreadsheet takes more time and effort.

Sorry (tm Canada 🇨🇦) but until issue #1933 is resolved, we need the numbers spreadsheet to look across the entire manifest in various ways.

No worries then!

xexyl commented 4 months ago

Also I have to say the text is rather small for my vision: it was quite hard to see. Now as for the vision problem an alternative is to increase the font size. Do you mind if I do this? It would make it much easier for me.

By all means, change the font size if that helps you see the data better.

Thanks .. will do that next time.

xexyl commented 4 months ago

I am also contemplating writing a script that automates some of the above. It would check the exit codes, run git diff, telling them to look at it very carefully and then after git diff exits it will prompt them if everything is okay. If they input anything but y or Y it will tell them to start over and then try again. It would also first ask them (before doing anything .. after parsing args I guess but maybe even before to save some time) if they are sure the manifest files are up to date.

If you feel the need to do this, please put such tool under tmp/. This process is temporary, until this issue #1933 is resolved.

That is indeed where I would put it. Thanks. We'll see if I can be bothered!

xexyl commented 4 months ago

I am going to set a reminder for every Saturday. If you wish me to do it twice a week I can do it say every Saturday and every Wednesday. I will set the reminder to 0400 so t hat I am sure to see it earlier on which means it'll be one of the first things I do at the repo. I believe this is a better way to handle it.

Twice a week should be OK. From experience, once you do it a few times it should become rather easy and manageable.

Is once a week okay too? I agree you're probably right on the last part though. It's just a lengthy process and it takes more time. I'd like to get the entries done too and if I only had to do it once a week the other issue would be quicker. If however you prefer it twice a week I'll see about doing that instead. Just let me know.

lcn2 commented 4 months ago

Is once a week okay too?

Sure.

xexyl commented 4 months ago

Is once a week okay too?

Sure.

Thanks! Reminder set already.

xexyl commented 4 months ago

Working on this now .. unfortunately been awake since before 2 :( .. once I am done (or maybe when running the html generation .. too tired to bother with just updating individual entries) I will try and rest. Doubt I can sleep again.

I decided to make a change that should help with this namely that I made the clean clone (no other files) the primary clone and the one with other files another directory name. Hopefully that will help speed things up (though slightly) and prevent any errors from occurring.

... oh, generating the json files takes a long while too ...naturally, now I thin on it, since it runs it on all entries ...

xexyl commented 4 months ago

Working on this now .. unfortunately been awake since before 2 :( .. once I am done (or maybe when running the html generation .. too tired to bother with just updating individual entries) I will try and rest. Doubt I can sleep again.

I decided to make a change that should help with this namely that I made the clean clone (no other files) the primary clone and the one with other files another directory name. Hopefully that will help speed things up (though slightly) and prevent any errors from occurring.

... oh, generating the json files takes a long while too ...naturally, now I thin on it, since it runs it on all entries ...

This week's task done! Going to try resting soon .. might be a slow day because of waking up at stupid o'clock but we'll see.

xexyl commented 4 months ago

Hmm .... I think that this might be a mistake?

        {   
            "file_path" : "index.html",
            "inventory_order" : 10,
            "OK_to_edit" : false,
            "display_as" : "html",
            "display_via_github" : false,
            "winners_text" : "winner home page"
        },

and in particular the winners_text. This is not the home page. Rather it is the winning entry README.md file generated as html. So what should it say instead? I'm too tired to consider it right now. Off to do other things but I wanted to bring this up.

lcn2 commented 4 months ago

Hmm .... I think that this might be a mistake?
        {   

            "file_path" : "index.html",

            "inventory_order" : 10,

            "OK_to_edit" : false,

            "display_as" : "html",

            "display_via_github" : false,

            "winners_text" : "winner home page"

        },  
and in particular the winners_text. This is not the home page. Rather it is the winning entry README.md file generated as html. So what should it say instead? I'm too tired to consider it right now. Off to do other things but I wanted to bring this up.

Winner index page
Index page for IOCCC winner
IOCCC winner index page
?

xexyl commented 4 months ago

Hmm .... I think that this might be a mistake?
        {   

            "file_path" : "index.html",

            "inventory_order" : 10,

            "OK_to_edit" : false,

            "display_as" : "html",

            "display_via_github" : false,

            "winners_text" : "winner home page"

        },  
and in particular the winners_text. This is not the home page. Rather it is the winning entry README.md file generated as html. So what should it say instead? I'm too tired to consider it right now. Off to do other things but I wanted to bring this up.
Winner index page

Index page for IOCCC winner

IOCCC winner index page

?

Hmm .. not the first one but either of the second and third could work. I'm not sure. I think the second one is more clear or more complete and less ambiguous. It might be even: ' Index page for IOCCC winning entry' or even ' Index page for an IOCCC winning entry'.

Or it could be the third one like: `IOCCC winning entry index page'. I actually think that sounds better of all of them. It's clear and a complete description (sentence, maybe, though one might not start a sentence with IOCCC as such but you know what I mean I guess). What do you think?

lcn2 commented 4 months ago

Hmm .... I think that this might be a mistake?
        {   

            "file_path" : "index.html",

            "inventory_order" : 10,

            "OK_to_edit" : false,

            "display_as" : "html",

            "display_via_github" : false,

            "winners_text" : "winner home page"

        },  
and in particular the winners_text. This is not the home page. Rather it is the winning entry README.md file generated as html. So what should it say instead? I'm too tired to consider it right now. Off to do other things but I wanted to bring this up.
Winner index page

Index page for IOCCC winner

IOCCC winner index page

?
Hmm .. not the first one but either of the second and third could work. I'm not sure. I think the second one is more clear or more complete and less ambiguous. It might be even: ' Index page for IOCCC winning entry' or even ' Index page for an IOCCC winning entry'.

Or it could be the third one like: `IOCCC winning entry index page'. I actually think that sounds better of all of them. It's clear and a complete description (sentence, maybe, though one might not start a sentence with IOCCC as such but you know what I mean I guess). What do you think?

Likely NOT 'IOCCC winning entry index page' because of the word entry. See below.

winner vs entry

While we sometimes fail to do so, we try to distinguish an "entry" (also called a submission) from a "winner" (as in something that won the IOCCC). Thus goes form the practice of only acknowledging the existence of IOCCC winners (as acknowledgment is in the only thing a winner receives) and not any entry that failed to win.

Maybe this practice was a mistake, but that's what we have used. Yet "winning entry" seems an awkward phrase, and we try to use "winner" when it comes to referring to some code that won the IOCCC, and "author(s)" to refer to those who are the author of a "winner".

In "long form", perhaps "winner" can be extended to "IOCCC winner", or something better (that doesn't use the word entry, nor submission, nor some word with a similar root)?

In "long form", "author" can be extended to "People who have won", or something better?

Now is probably the time to go thru text/code/documentation to refer to "winner" and not "entry" (in short or long form).

xexyl commented 4 months ago

Hmm .... I think that this might be a mistake?
    {   
        "file_path" : "index.html",
        "inventory_order" : 10,
        "OK_to_edit" : false,
        "display_as" : "html",
        "display_via_github" : false,
        "winners_text" : "winner home page"
    },  
and in particular the winners_text. This is not the home page. Rather it is the winning entry README.md file generated as html. So what should it say instead? I'm too tired to consider it right now. Off to do other things but I wanted to bring this up.
Winner index page

Index page for IOCCC winner

IOCCC winner index page

?
Hmm .. not the first one but either of the second and third could work. I'm not sure. I think the second one is more clear or more complete and less ambiguous. It might be even: ' Index page for IOCCC winning entry' or even ' Index page for an IOCCC winning entry'. Or it could be the third one like: `IOCCC winning entry index page'. I actually think that sounds better of all of them. It's clear and a complete description (sentence, maybe, though one might not start a sentence with IOCCC as such but you know what I mean I guess). What do you think?
Likely NOT 'IOCCC winning entry index page' because of the word entry. See below.

Okay it won't be that then. Not a problem.

winner vs entry

While we sometimes fail to do so, we try to distinguish an "entry" (also called a submission) from a "winner" (as in something that won the IOCCC). Thus goes form the practice of only acknowledging the existence of IOCCC winners (as acknowledgment is in the only thing a winner receives) and not any entry that failed to win.

I wish I had known that earlier though it makes sense in a way.

Maybe this practice was a mistake, but that's what we have used. Yet "winning entry" seems an awkward phrase, and we try to use "winner" when it comes to referring to some code that won the IOCCC, and "author(s)" to refer to those who are the author of a "winner".

It might have been a mistake but more important to be consistent. I don't think of 'winning entry' as awkward but if you do then that's what matters.

In "long form", perhaps "winner" can be extended to "IOCCC winner", or something better (that doesn't use the word entry, nor submission, nor some word with a similar root)?

Please elaborate on what you mean by 'long form' and also when it should be used versus not. What about short form (same questions)?

I think IOCCC winner does sound nice but the reason (my rationale) I put in 'entry' is because there are many winners including each year so which one is it? Also does it refer to the person (author) or the entry itself? That's why I put in entry in places I did (and see below on that).

In "long form", "author" can be extended to "People who have won", or something better?

Isn't this already the case in the main page of the website ? 'Author' indicates a single person whereas 'people who have won' suggests all winners. That's also my concern: the two are not the same thing.

Now is probably the time to go thru text/code/documentation to refer to "winner" and not "entry" (in short or long form).

I was thinking the same thing but I think probably discussion is in order first (as I'm not sure what you mean by long or short form and when each should be used).

Thanks. I feel very fatigued today so I don't know how much I'll get done. I do hope to get some done though. We'll see.

lcn2 commented 4 months ago

In "long form", perhaps "winner" can be extended to "IOCCC winner", or something better (that doesn't use the word entry, nor submission, nor some word with a similar root)? Please elaborate on what you mean by 'long form' and also when it should be used versus not. What about short form (same questions)?

Two ways to express the same thing, shorter and longer.

xexyl commented 4 months ago

In "long form", perhaps "winner" can be extended to "IOCCC winner", or something better (that doesn't use the word entry, nor submission, nor some word with a similar root)? Please elaborate on what you mean by 'long form' and also when it should be used versus not. What about short form (same questions)?

Two ways to express the same thing, shorter and longer.

Sure but where should each go?

lcn2 commented 4 months ago

In "long form", perhaps "winner" can be extended to "IOCCC winner", or something better (that doesn't use the word entry, nor submission, nor some word with a similar root)? Please elaborate on what you mean by 'long form' and also when it should be used versus not. What about short form (same questions)?

Two ways to express the same thing, shorter and longer.

Sure but where should each go?

It depends on how much space you have, on if you are repeating the content (say 1st time in long form, 2nd time in short form), context, use of judgment, etc.

xexyl commented 4 months ago

In "long form", perhaps "winner" can be extended to "IOCCC winner", or something better (that doesn't use the word entry, nor submission, nor some word with a similar root)? Please elaborate on what you mean by 'long form' and also when it should be used versus not. What about short form (same questions)?

Two ways to express the same thing, shorter and longer.

Sure but where should each go?

It depends on how much space you have, on if you are repeating the content (say 1st time in long form, 2nd time in short form), context, use of judgment, etc.

Okay the next question is: what should the long form be and what should the short form be? I want to make sure it's right before I try and fix any changes (though again to me it makes less sense to have 'IOCCC winner' when referring to an entry as 'IOCCC winner' implies the author). But at least in the try scripts I have in the form of ...

try.sh - demonstrate IOCCC winner YYYY/author
try.alt.sh - demonstrate IOCCC winner YYYY/author alt code

so presumably nothing has to be done with those. Is that correct? Though now I think on it with other scripts like author.sh it might need to have better header comments ...

lcn2 commented 4 months ago

We agree that there is some inherent ambiguity for the term "winner".

Let's put it this way: IOCCC judges review programs when judging a new IOCCC. We go out of our way to NOT KNOW who submitted a program during the judging process.

When the winners of a new IOCCC are announced, it is the programs that are announced as having won. Only at the very end, just before the announcement do the judges discover who are the authors of the winning programs.

And an author can have more than one winning entry of an IOCCC.

So that's the origin of "IOCCC winners" as the programs, and "IOCCC authors" as the person(s) who submitted winning programs.

Short forms: winner 🏆 and author ✍️

Longer forms: TBD ..

.. but perhaps "IOCCC winner" and/or "winning IOCCC program" and perhaps "IOCCC author" and "author of an IOCCC winner".

We probably need to think this out some more before we consider creating a new issue (one that wouldn't be a top priority) to address the use of these terms, make their use consistent across the IOCCC site, and add an FAQ about them.

UPDATE 0

The short form terms winner 🏆 and author ✍️ can continue to be used now.

The term "entry" can be avoided for now.

The term "submission", for now, can refer to a program (packaged by mkiocccentry) submitted to the IOCCC judges for consideration.

The possible new lower priory issue can be where a change to better terms, and possible long forms of such better terms, and an FAQ entry about the terms can be created.

There are not a huge number of places where the terms are used. By using short form terms winner 🏆 and author ✍️ now, the result of the new issue is a change of terms then it would be that hard to change them (if needed).

xexyl commented 4 months ago

We agree that there is some inherent ambiguity for the term "winner".

That's good. Though to be fair it might also be me being too literal in some ways.

Let's put it this way: IOCCC judges review programs when judging a new IOCCC. We go out of our way to NOT KNOW who submitted a program during the judging process.

Yes as the guidelines and rules state.

When the winners of a new IOCCC are announced, it is the programs that are announced as having won. Only at the very end, just before the announcement do the judges discover who are the authors of the winning programs.

That's true - an interesting way of looking at it.

And an author can have more than one winning entry of an IOCCC.

True.

So that's the origin of "IOCCC winners" as the programs, and "IOCCC authors" as the person(s) who submitted winning programs.

Short forms: winner 🏆 and author ✍️

Longer forms: TBD ..

.. but perhaps "IOCCC winner" and/or "winning IOCCC program" and perhaps "IOCCC author" and "author of an IOCCC winner".

Could be. How things happen can be really interesting very often.

We probably need to think this out some more before we consider creating a new issue (one that wouldn't be a top priority) to address the use of these terms, make their use consistent across the IOCCC site, and add an FAQ about them.

That sounds like a great idea. I'll hold off then. Thanks.

lcn2 commented 4 months ago

In regards to this issue about the manifest, the _winnerstext is the only field where the terms discussed in comment 1919762844 might arise.

Nevertheless, the _winnerstext needs to be kept short. So perhaps only the short form terms of winner and author should be used for now.

After the new issue hinted at in comment 1919762844 is addressed can one go thru, and if needed change to using better terms.

xexyl commented 4 months ago

In regards to this issue about the manifest, the _winnerstext is the only field where the terms discussed in comment 1919762844 might arise.

Nevertheless, the _winnerstext needs to be kept short. So perhaps only the short form terms of winner and author should be used for now.

After the new issue hinted at in comment 1919762844 is addressed can one go thru, and if needed change to using better terms.

So is it okay to just hold off until then? Either way I'm too tired to fully parse this comment so I'll have to hold off until another time.

lcn2 commented 4 months ago

We are off of a cardiologist appointment. We will think about this matter and likely create an issue around it.

Of course, after we process all of the outstanding pull requests after our appointment as well.

xexyl commented 4 months ago

We are off of a cardiologist appointment. We will think about this matter and likely create an issue around it.

Of course, after we process all of the outstanding pull requests after our appointment as well.

Best wishes at the cardiologist! And thanks for the kind words. I didn't expect to get nearly as much done today but I'm happy I did. Tomorrow likely will be slower but then I'm so close to being done that I might finish the scripts anyway.

Anyway hope all's well with your heart!

lcn2 commented 4 months ago

We are off of a cardiologist appointment. We will think about this matter and likely create an issue around it. Of course, after we process all of the outstanding pull requests after our appointment as well.

Best wishes at the cardiologist! And thanks for the kind words. I didn't expect to get nearly as much done today but I'm happy I did. Tomorrow likely will be slower but then I'm so close to being done that I might finish the scripts anyway.

Anyway hope all's well with your heart!

routine checkup .. all is well, thanks 🙏

xexyl commented 4 months ago

We are off of a cardiologist appointment. We will think about this matter and likely create an issue around it.

Of course, after we process all of the outstanding pull requests after our appointment as well.

Best wishes at the cardiologist! And thanks for the kind words. I didn't expect to get nearly as much done today but I'm happy I did. Tomorrow likely will be slower but then I'm so close to being done that I might finish the scripts anyway.

Anyway hope all's well with your heart!

routine checkup .. all is well, thanks 🙏

I guessed and hoped for that. Glad to hear it!

You know it's funny but the heart is one of the organs that I simultaneously don't want to know about the problems that can happen and yet obviously everyone should.

I think what scares me is the electrophysiology studies. But it's a fascinating organ that obviously is vital to existence.

Did you know that they say that often people will feel a sense of impending doom for up to a month before a heart attack?

Did you also know that heartbreak can cause physical damage to the heart? (Or at least cause health problems.) Iirc it's called heart break syndrome. But that's a part that I am not too knowledgeable about, how heartbreak can cause physical problems. Well I can see how it can happen but I have only briefly read about it and not in detail.

Anyway glad all was well!

xexyl commented 4 months ago

As far as hearts go I think it's the perfect time to share one of my puns (that actually has more than one pun): many people claim that their heart is in the right place. But they're bloody liars; it's in the left place.

xexyl commented 4 months ago

I myself have actually had an interesting experience with the heart. Years ago either 2004 or 2005 I had thrown up in the morning. Later that day I had chest pain. Many people would pass it by as unimportant but I am not a fool.

So I went to hospital (emergency department). They did an enzyme test to check if I had a heart attack (even though it was very unlikely it was obviously the right thing to do to rule it out).

As expected no problem there. But what was totally unexpected is that my platelets were very low. In fact they were rapidly depleting. I was hospitalised for a few days.

Had to be on a corticoid for six months. It was awful. My mood was not exactly good at the time and corticoids as you might know make that even worse. But more annoying of course is if you have been on them long enough you have to taper off them. If you don't it can kill you.

I then had to have IVIG. It was supposed to be quick but I was there for hours. I seem to recall having gastrointestinal issues too.

But it's pretty interesting how for something totally unrelated that very serious problem was discovered.

Now the hospital staff wanted to take me off all my medications. Even if that would have been safe (and it absolutely would not have been) (and I mean physically safe but also I would have been a proper mess) there was no need. I knew immediately what caused it. I know my body very well!

I told them about the medication that I had been put on recently that I shouldn't have been put on in the first place. Non RA but that doctor put me on an immunosuppressant. That's what did it.

I was very lucky in a lot of ways. I hope that that I never have an actual problem with my heart in the future but it's entirely possible that I will sometime down the road. Many people do after all.

Unless of course the Vogons finally find us before that of course!

And now it's time for me to get some sleep.

Hope you have a great night!

xexyl commented 3 months ago

Just finished a script to do this. Will tell you more later .. have to go afk a while. Will use it to update manifest when back.

xexyl commented 3 months ago

Just finished a script to do this. Will tell you more later .. have to go afk a while. Will use it to update manifest when back.

Testing all paths (script flow I mean). Then if all good I'll do the commit and add the script though I guess it'll only be useful a few more times. But makes it easier than having to go back and forth between here and shell.

ioccc-src / temp-test-ioccc

Enhancement: Improve the IOCCC manifest #1933

TODO

Initial TODO list

Final TODO list

What is the IOCCC manifest?

Caution: numbers spreadsheets look like a single blob to git

How to verify that numbers manifest matches the entry file tree

Step -1

Step 0

Step 2

Step 2a

Step 2b

Step 2c

Step 2d

Step 3

Step 4

Step 5

Step 6

Update .entry.json files

Step 7

Step 8

Step 9

Update entry index.html files

Step 10

Step 11

Step 12

To canonically sort the manifest numbers spreadsheet

Manifest fields

year

dir

file_path

inventory_order

OK_to_edit

display_as

display_via_github

entry_text

UPDATE 0

ALTERNATIVELY

ALTERNATIVELY

ALTERNATIVELY

UPDATE 0

UPDATE 0

UPDATE 0

UPDATE 1

UPDATE 0

UPDATE 0

winner vs entry

winner vs entry

UPDATE 0

Update `.entry.json` files

Update entry `index.html` files