search

ioccc-src / temp-test-ioccc

Temporary test IOCCC web site that will go away
Creative Commons Attribution Share Alike 4.0 International
48 stars 7 forks source link

Enhancement: Perform the Great Fork Merge #2239

Open lcn2 opened 8 months ago

lcn2 commented 8 months ago

The Great Fork Merge

The Great Fork Merge will occur when the multi-thousand commits that the temp-test-ioccc repo is ahead of the Official IOCCC winner repo are bright back to main Official www.ioccc.org web site.

TODOs

In order to perform the Great Fork Merge, the following tasks need to be completed:

NOTE: This may require a rewrite of the bin/ioccc-status.sh tool as well change the format of the from status.json file.

Port the bin tools tools to run under RHEL 9 version of Linux. With this port, the IOCCC judges should be able to use these tools on a wide enough variety of systems for their purposes.

See also comment 1993767060.

See comment 9192931.

The exception, of course, is the initial text down to the XXX that indicates this is a test repo.

Improve the look and feel of the constructed HTML files.

Reorganize the FAQ.

Prepare the mkiocccentry repo for a code freeze and release.

See also GH-issue 931 from that other repo.

NOTE: JSONPath.sh with only -S -A and without -T may be helpful with this coversion.

NOTE: The bin/jprint-wrapper.sh tool should also be updated.

This will be in keeping with the completed TODO the JSON indented with multiples of 4 ASCII spaces and no TABs.

Now that man pages for bin/ tools (issue #2009) are not planned, we need to review and likely improve the bin/README.md documentation.

We need to add some notes about these recent additions:

Issue #2686 was created to satisfy this TODO item.

See comment-2189364148.

See comment 919283.

See comment-2198635072.

This TODO was changed from referring to a "spoiler" into a "de-obfucated" educational emphasis. In that regard we should NOT use terms like "spoiler" in the FAQ as well as NOT using "spoiler" in the entry's README.md file.

At a most, README.md should suggest that the reader might wish to study the prog.c code first before going on to review the de-obfuscated "alt" code.

See also comment-2198807715.

After completing the FAQ entry how to handle de-obfuscated code, consider if needed, revising entries with existing de-obfuscated code AND if needed, fixing cases where the entry README.md makes reference to something being a spoiler.

See comment-2172105426 and see comment-2174364721.

See comment-2197371351.

Add a "see also" link to the new FAQ 1.4 in FAQ 0.0 subsection 2.

If the registration process is not "screenshot ready" by the time this TODO is addressed, put in a "stay tuned for images" placeholder into this new FAQ.

Add a "see also" link to the new FAQ 1.4 in FAQ 0.0 subsection 5.

If the submit process is not "screenshot ready" by the time this TODO is addressed, put in a "stay tuned for images" placeholder into this new FAQ.

In addition to the useful check for typos, wording, and broken links: review for suitability for gong live with on the main web site after the Great Fork Merge. Update faq.md if and as needed.

We need to cleanup from testing tools, so we need to remove:

These may have added as part of tool testing and need to be removed before we proceed along the Great Fork Merge process below.

Add, modify and/or remove things from the remaining TODOs if/as needed.

In a comment in this issue #2239, announcers that general pull requests for the temp-test-ioccc repo will no be accepted and that we are beginning work on the Near Final TODOs.

Getting ready for the Near Final TODOs

NOTE: Once the above TODOs have been completed, the following last minute TODO actions must be completed in relatively short order (preferably in the same calendar day) before the Great Fork Merge takes place.

Use this to look for any whitespace from markdown files

find . -name '*.md' -print0 | xargs -0 grep -E -l '[[:space:]][[:space:]]*$'

Use:

find . -name '*.md' ! -name markdown.md -print0 | xargs -0 pcregrep -M '^``.*\n[^|`\n \t]'

After making sure that the temp-test-ioccc repo is up to date and related GitHub pages have been rendered, use the ✓ on the navbar to check all generated HTML pages.

Fix any errors, warnings and info messages reported. Update the temp-test-ioccc repo and recheck those pages.

In FAQ 1.2, look for the <!-- XXX - Fill in the date when Great Fork Merge happens --> and update the date as for that section as needed.

Change default URLs and REPOs to refer to the https://www.ioccc.org web site and winner repo.

Change references as needed given their context.

This includes markdown files, HTML files, var.mk, Makefiles, text files, etc.

Try:

make www
find . \( -name .git -o -name NOTES \) -prune -o -type f -print0 | xargs -0 grep -l -F temp-test-ioccc

Also change the comment in var.mk

See GH-#issuecomment-2403175498.

The only exceptions should be historic references found in news.md, news.hml, faq.md and faq.html.

Edit news.md as needed.

Perform make www and commit any changes.

Near Final TODOs

NOTE: Once the above TODOs have been completed, the following last minute TODO actions must be completed in relatively short order (preferably in the same calendar day) before the Great Fork Merge takes place.

Be sure the mkiocccentry repo is up to date and perform a make install in that typo.

See GH-issuecomment-2430718370 for a summary of the test process.

We will perform the following actions:

git mv 2011/dlowe/dlowe-aux-data.tar.bz2 HOLD
git commit -m'move dlowe-aux-data.tar.bz2 to HOLD'

git remote -v
git filter-repo --path-glob '????/*.tar.bz2' --invert-paths --force
git remote add origin git@github.com:lcn2/trim-temp-test-ioccc.git

git mv HOLD 2011/dlowe/dlowe-aux-data.tar.bz2
git commit -m'move HOLD back to dlowe-aux-data.tar.bz2'

for year in [12][0-9][0-9][0-9]; do
  touch "$year/$year.tar.bz2"; git add "$year/$year.tar.bz2";
done

for year in [12][0-9][0-9][0-9]; do
  for dir in $(< "$year/.year"); do
    entry_id=${dir/\//_};
    touch "$dir/$entry_id.tar.bz2";
    git add "$dir/$entry_id.tar.bz2";
  done;
done

git commit -m'add empty tar.bz2 files until we do make update'

git repack
git prune-packed
git reflog expire --expire=now --all
git gc --aggressive --prune=now
git fsck --full

git push --all --force
git push --tags --force
git push --set-upstream origin master

FYI: See:

NOTE: The make update in a TODO farther below will add back the needed *.tar.bz2 files.

NOTE: The make gen_years will generate Warnings of the form: "bin/gen-years.sh: Warning: compressed tarball for YYYY: yyyy is empty: yyyy/yyyy.tar.bz2".

This is OK and will be corrected once make update or make tar is done.

If so, then revert those changes by:

git restore sitemap.xml status.html status.json

If not, reset all TODOs under the Near Final TODOs section and fix the cause.

If not, reset all TODOs under the Near Final TODOs section and fix the cause.

Remove the lines between:

<!-- XXX - This entire section goes away during the final stages of the Great Fork Merge -->

and:

<!-- XXX - remove down to here in the final stages of the Great Fork Merge -->

Verify that:

git status

reports:

On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

Fetch any last minute changes:

cd docroot/temp-test-iocc

git fetch && git rebase

git clean -f

git status

The last command should report:

On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
git commit -m'final pre-Great Fork Merge'

git push

Fetch any last minute changes:

cd docroot/temp-test-iocc

git fetch && git rebase

git clean -f

git status

The last command should report:

On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

If so, then revert those changes by:

git restore sitemap.xml status.html status.json

If not, reset all TODOs under the Near Final TODOs section and fix the cause.

If not, reset all TODOs under the Near Final TODOs section and fix the cause.

FINAL REVIEW

We expect this period to be between 2024 November 1 and 2024 November 15.

Please review the effect of all of the above completed to do. Please limit PRs to only critical problems, and broken links, and linked to the wrong file, and failure to the official website and repo, etc.

Accept any such PRs, if needed modify the result after accepting, and finally do a make update.

Great Fork Merge

Click Create Pull Request for the Official IOCCC winner repo.

Inspect the pull request for the Official IOCCC winner repo.

Accept and complete the pull request.

Post Great Fork Merge TODOs

Once the Great Fork Merge occurs and the official IOCCC winner repo and related Official www.ioccc.org web site has been updated, these TODOs need to be performed on the official IOCCC winner repo:

Close down issue #2686 as "won't fix" in this repo.

Fix any problems found on the Official www.ioccc.org web site by editing the official IOCCC winner repo as needed.

From temp-test-ioccc repo settings click on the ((Archive this repository)) button.

This will Freeze the temp-test-ioccc repo but leave it in place for historic purposes.

xexyl commented 8 months ago

Please assign this to me.

And as I noted before: if needed I would like to update my script that updates the status.json file.

Thanks.

I will have to look at this issue later on today as it seems rather important!

xexyl commented 8 months ago

Also I am not sure if I can finish the year README files before the 10th but I should be able to finish them sometime soon anyway.

But for now I must go do other things again. Back in a while.

xexyl commented 8 months ago

In order to perform the Great Fork Merge, the following tasks need to be completed:

  • Revise status.json format.
  • Write a new tool to generate the new top level status.html from status.json

NOTE: This may require a rewrite of the bin/ioccc-status.sh tool as well change the format of the from status.json file.

It's funny you should say that .. the generating the file. Do you mean creating the file rather than modifying it? I actually had the thought of adding an option to the script to do that and I seem to recall that I almost did do it. It would be easy to do. But since the format is not necessarily the final format I will hold off on that.

The anticipated completion date for this issue is 2024 March 10 2:30 (2:30AM) Pacific.

Like I said over there mostly it is done: except for the YYYY/README.md files. But I can do those after 10 March if necessary so I have no problem with it being closed. You were going to take care of the silencing the warnings and right now would be fine as I don't anticipate modifying any Makefiles or if I'm honest for now anything at all (except maybe the FAQ which you asked me to look at). That probably will hold for a day or two at least but even if I do make other changes it'll likely only be the YYYY/README.md files. I say I might not finish it before 10 March but I might yet: I just have to have the energy and time and the former seems to be a problem lately I'm afraid. Even so I might be able to do some of it today: that remains to be seen however.

Okay in order to do this I think I even created a new repo to test how this works. I think the other issues should come first though. Is that correct?

Ah yes ... I forgot. I have to update descriptions and also check the html files! It's been very hectic and I totally forgot about this. I was also holding off as you were planning many changes that would conflict. Those changes are now done so I can look at this sometime soon too. If I don't please remind me.

Right ... well this one might be good to look at when I look at the manifest too. After all the result of the manifest is in the index.html files.

Well some of this cannot be done until the submit server is ready, right?

Near Final TODOs

Once the above TODOs have been completed, the following last minute TODO actions must be completed in relatively short order before the Great Fork Merge takes place:

In the (few?) cases where a absolute URL is required, change that URL to be under https://www.ioccc.org and NOT this repo's top level URL.

sgit(1) should be useful for this one! :)

Edit news.md as needed.

That's a great idea .. looking forward to see what you have in mind! Almost eager enough to try and get myself to work on YYYY/README.md files today but I think that might be a mistake with how tired I feel :(

  • Verify that the installed location(1) tool is up to date

How do you mean verify that it's up to date? Do you mean locally installed? But what version should it be then? Did you have other updates in mind? (I in any case have the most recent version at the other repo.) Then again perhaps this is something you have to do so I might not have to worry about it. Or maybe a tool here uses it and I do have to worry about it. But as noted I do have them installed so it's fine.

  • Verify manifest.numbers agrees with the local temp-test-ioccc repo

How do you mean verify that it agrees with it when it's in the repo? Though again I guess this might have to be something you do so I don't need to worry about it: updating the manifest I do have to worry about but perhaps not this step?

Perform the 12 steps as noted in comment 2089758002.

Don't forget the script I wrote! It's very useful. By default it does not commit automatically but there is an option to allow it to do so. It even asks if you're on the right branch.

  • Remove tmp directory.

Ever notice what tmp spells backwards? :-)) Well okay it is a British English abbreviation but still :-)

Post Great Fork Merge TODOs

Once the Great Fork Merge occurs and the official IOCCC winner repo and related Official www.ioccc.org web site has been updated, these TODOs need to be performed on the official IOCCC winner repo:

I've enjoyed this one though I must say! A lot of good things, a lot of fun, a lot of laughs and stories and other things ... thanks for the privilege and honour!

Good idea.

  • Mark the temp-test-ioccc repo read-only

From temp-test-ioccc repo settings click on the ((Archive this repository)) button.

THANK YOU for going for that (as I suggested) rather than deleting it.

  • Verify site links work Ah yes .. that'll be fun! :-)

Process TBD.

Fix any problems found as needed.

Of course.

At the other repo I left a comment regarding an issue I have (about finishing something we talked about the other day) but I think that and this comment is probably all I'll manage to do today for the contest. But who knows. I might be able to take a look at some things at this repo too. For now I'll be afk (or I intend to be afk soon). I think today though I will mostly be reading when not doing the things I have to do.

lcn2 commented 8 months ago

Incomplete TODO

NOTICE: This TODO list will be built over a few days. Until this notice is removed, consider this list to be incomplete.

:-)

=-=

  • Revise status.json format.
  • Write a new tool to generate the new top level status.html from status.json

NOTE: This may require a rewrite of the bin/ioccc-status.sh tool as well change the format of the from status.json file. It's funny you should say that .. the generating the file. Do you mean creating the file rather than modifying it?

The content of status.json is still TBD, and so is the tool that will mange it.

=-=

The anticipate completion date for this issue is 2024 March 10 2:30 (2:30AM) Pacific.

Like I said over there mostly it is done: except for the YYYY/README.md files. But I can do those after 10 March if necessary so I have no problem with it being closed.

Sure.

BTW: issue #2006 does indicate that it covers such files, and the title has been changed to say "constructed HTML files".

lcn2 commented 8 months ago

Okay in order to do this I think I even created a new repo to test how this works. I think the other issues should come first though. Is that correct?

It is a ((*top priority)) issue and has been for a while. It pertains to something that is much needed after the Great Fork Merge when people start considering opening up issues on the revised web site.

=-=

Well some of this cannot be done until the submit server is ready, right?

We will need to put in some placeholder text that indicates the exact process has not been created, but will be before the IOCCCMOCK is held.

=-=

  • Verify that the installed location(1) tool is up to date

How do you mean verify that it's up to date? Do you mean locally installed? But what version should it be then? Did you have other updates in mind? (I in any case have the most recent version at the other repo.) Then again perhaps this is something you have to do so I might not have to worry about it. Or maybe a tool here uses it and I do have to worry about it. But as noted I do have them installed so it's fine.

As we need to be the one be performing these last steps, you @xexyl do not need to worry about the details.

=-=

  • Remove tmp directory.

Ever notice what tmp spells backwards? :-)) Well okay it is a British English abbreviation but still :-)

:-)

xexyl commented 8 months ago

Okay in order to do this I think I even created a new repo to test how this works. I think the other issues should come first though. Is that correct?

It is a ((*top priority)) issue and has been for a while. It pertains to something that is much needed after the Great Fork Merge when people start considering opening up issues on the revised web site.

Let me reword that. Besides #3 (and I hope I can look at some of it today in a while but we shall see) do you have an order of priority for the remaining issues or should it be 'whatever I have the time and energy and inclination for'?

That idea of whatever I can work on would be useful but if you have an order I could try and work with it.

=-=

Well some of this cannot be done until the submit server is ready, right?

We will need to put in some placeholder text that indicates the exact process has not been created, but will be before the IOCCCMOCK is held.

That all makes sense to me yes. Thanks.

=-=

  • Verify that the installed location(1) tool is up to date

How do you mean verify that it's up to date? Do you mean locally installed? But what version should it be then? Did you have other updates in mind? (I in any case have the most recent version at the other repo.) Then again perhaps this is something you have to do so I might not have to worry about it. Or maybe a tool here uses it and I do have to worry about it. But as noted I do have them installed so it's fine.

As we need to be the one be performing these last steps, you @xexyl do not need to worry about the details.

I guessed that but I wanted to be sure.

=-=

  • Remove tmp directory.

Ever notice what tmp spells backwards? :-)) Well okay it is a British English abbreviation but still :-)

:-)

:-) :-)

lcn2 commented 8 months ago

Let me reword that. Besides https://github.com/ioccc-src/temp-test-ioccc/issues/3 (and I hope I can look at some of it today in a while but we shall see) do you have an order of priority for the remaining issues or should it be 'whatever I have the time and energy and inclination for'?

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .

We home this clarifies @xexyl .

We will be mostly offline for about 2-3 days working on issue #2008 as well as prep actions for this issue.

xexyl commented 8 months ago

Let me reword that. Besides #3 (and I hope I can look at some of it today in a while but we shall see) do you have an order of priority for the remaining issues or should it be 'whatever I have the time and energy and inclination for'?

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 .

I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

We home this clarifies @xexyl .

Hopefully :-) it does .. yes it does - thanks!

We will be mostly offline for about 2-3 days working on issue #2008 as well as prep actions for this issue.

Best wishes! Today I'm not doing anything for the contest I'm afraid. Having a very off day. Tomorrow i'm sure will be better.

lcn2 commented 8 months ago

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 . I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

Correct, @xexyl

xexyl commented 8 months ago

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 . I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

Correct, @xexyl

Thanks. I want to be sure of something before I finish my part of #3 which I'll ask you there. My part left in that is minimal except of course for checking typos etc. in the FAQ and other like files so that's good. But I won't be doing anything today. Depending on when you get back to me on the concern and the answer too I hopefully can start tomorrow.

lcn2 commented 8 months ago

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 . I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

Correct, @xexyl

Thanks. I want to be sure of something before I finish my part of #3 which I'll ask you there. My part left in that is minimal except of course for checking typos etc. in the FAQ and other like files so that's good. But I won't be doing anything today. Depending on when you get back to me on the concern and the answer too I hopefully can start tomorrow.

Best wishes, @xexyl

xexyl commented 8 months ago

After closing issue #3 , the next priorities will be issue #1933, then issue #2006 , then issue #858 . I presume that this is in order and not equal priority - given that you say 'then' and 'then'?

Correct, @xexyl

Thanks. I want to be sure of something before I finish my part of #3 which I'll ask you there. My part left in that is minimal except of course for checking typos etc. in the FAQ and other like files so that's good. But I won't be doing anything today. Depending on when you get back to me on the concern and the answer too I hopefully can start tomorrow.

Best wishes, @xexyl

Should be good: I'm going to soon get back to LR! And you too!

lcn2 commented 8 months ago

With commit 449205e1a50f6e2dad5aa22301d965689059c885 the new status.json model is complete.

The first 2 TODO items of this issue #2239 have been completed.

lcn2 commented 8 months ago

We will next focus on issue #2008 as that issue is somewhat independent of the other ((Top Priority)) issues.

lcn2 commented 8 months ago

We will next focus on issue #2008 as that issue is somewhat independent of the other ((Top Priority)) issues.

Issue #2008, as per comment 1987654095 and commit f482380661b88ca3e0f4a70c4d9b4d5ccb17e9f0 has been completed.

lcn2 commented 8 months ago

We made more tweaks to the TODO list for this issue.

We think the TODO list is nearly ready.

UPDATE 0

Comments and corrections welcome.

lcn2 commented 8 months ago

We believe we have addressed all of the current questions that still need answering at this time. If we've missed something or something else needs to be clarified, please ask again.

xexyl commented 8 months ago

We believe we have addressed all of the current questions that still need answering at this time. If we've missed something or something else needs to be clarified, please ask again.

I will have to get back to this tomorrow and determine that. I'm sure you did though.

lcn2 commented 8 months ago

With commit 4471875497ed9ed050aa8e9bb736ac87cc55dd31 the bin tools have been ported to RHEL 9.

The make quick_www and make www should work well under Linux, assuming that up to date critical tools such as location(1), jparse(1) are installed and assuming that pandoc(1) has ben installed with a minimum 3.1.12.2 version, and when bash(1) found by #!/usr/bin/env bash is version 5.1.8 or later.

UPDATE 0

The TODO list has been updated accordingly.

xexyl commented 8 months ago

With commit 4471875 the bin tools have been ported to RHEL 9.

The make quick_www and make www should work well under Linux, assuming that up to date critical tools such as location(1), jparse(1) are installed and assuming that pandoc(1) has ben installed with a minimum 3.1.12.2 version, and when bash(1) found by #!/usr/bin/env bash is version 5.1.8 or later.

UPDATE 0

The TODO list has been updated accordingly.

Well now I ran into a new problem. Don't you use pandoc from Homebrew? If so: I updated it yesterday and it's not a high enough version for running the make www. Where do you get it from then? Thanks.

xexyl commented 8 months ago

With commit 4471875 the bin tools have been ported to RHEL 9. The make quick_www and make www should work well under Linux, assuming that up to date critical tools such as location(1), jparse(1) are installed and assuming that pandoc(1) has ben installed with a minimum 3.1.12.2 version, and when bash(1) found by #!/usr/bin/env bash is version 5.1.8 or later.

UPDATE 0

The TODO list has been updated accordingly.

Well now I ran into a new problem. Don't you use pandoc from Homebrew? If so: I updated it yesterday and it's not a high enough version for running the make www. Where do you get it from then? Thanks.

Oh I know why. I have it from MacPorts too. My guess is it'll be a problem to remove it so I have to maybe put the Homebrew path before the MacPorts path.

UPDATE 0

All good it seems. Thanks. I can now look at the comments in the other issue and hopefully get them done in the next few days. Right now though I must go again. It is unclear to me yet if I can do much today or anything at all but at least last night was better than the previous three nights! Hopefully that continues.

lcn2 commented 8 months ago

Priorities

Regarding the top priory open issues:

While going thru the IOCCC manifest, reviewing the data fields from the manifest.numers spreadsheet:

Look at each entry's inventory, as found in index.html#inventory to see how the contents are being expressed. Look at the sorted lines in the manifest.numers spreadsheet that relate to the entry in question.

Because issue #1933 and issue #2006 are somewhat intertwined @xexyl, we suggest you process both them at the same time. Like you did for issue #3, you can announce your progress on an IOCCC year by YEAR basis.

Regarding issue #2006

And since one is looking a an entry's inventory, give a glance over the rest of the index.html page.

Here are the types of questions, @xexyl, to ask yourself, does the index.html page contain:

We are NOT focusing on the accuracy of the index.html page per se (although if you see something glaring, feel free to fix it under the guide of something that slipped passed the issue #3 effort). We are focusing on HOW it LOOKS with an emphasis on correcting dysfunctional presentations.

In nearly all cases, corrections can be addressed by editing the entry's README.md file. Please modify the entry's README.md file and re-build the entry's index.html in a pull request.

If, however, you see an issue that may be more a matter of changing ioccc.css or the related bin tool, please raise the matter as a comment in this issue #2239 so we can discuss the matter before editing the CSS and/or bin tool. Again, this is less likely to happen, but if it does, we recommend a comment related discussion first. Modification to the entry's README.md file can be done directly via pull requests.

Regarding issue #1933

Here are the types of questions, @xexyl, to ask yourself:

See the _entrytext field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

See the _inventoryorder field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

See the _inventoryorder field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

See the _OK_toedit field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

See the _OK_toedit field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

See the _displayas field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

See the _display_viagithub field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

See the _display_viagithub field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

See the _display_viagithub field in the manifest.numers spreadsheet.

See also the Manifest fields section of the top comment of issue 1933.

See the Manifest fields section of the top TODO comment for issue #1933 for more information.

Comments, Suggestions as Questions welcome

Any Comments, Suggestions as Questions you may have about this comment are welcome.

lcn2 commented 8 months ago

As per comment 2010767087, this TODO item has been marked complete:

lcn2 commented 8 months ago

The results of test of make update may be found in commit 8aefcaa569b456ad501a5a4eafb3f18ea68ac10e

As it was sometimes said over the radio in the past: "This is only a test ... beeeeeeeeeeeep!" :-)

The test was successful under macOS.

NOTE: The make update action should only be done by IOCCC judges.

lcn2 commented 8 months ago

When pushing commit 8aefcaa569b456ad501a5a4eafb3f18ea68ac10e where the top level ioccc.tar.bz2 everything comprised tarball is formed, GitHub issues the following warnings:

remote: warning: See https://gh.io/lfs for more information.
remote: warning: File ioccc.tar.bz2 is 62.02 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.

So far nothing bad seems to happen, however we probably should use the Git Large File Storage for compressed tarballs?

UPDATE 0

We are looking into using Git Large File Storage for the compressed tarballs.

We will study how to use git lfs migrate as well.

We do NOT know if this is a good idea or not.

xexyl commented 8 months ago

When pushing commit 8aefcaa where the top level ioccc.tar.bz2 everything comprised tarball is formed, GitHub issues the following warnings:

remote: warning: See https://gh.io/lfs for more information.
remote: warning: File ioccc.tar.bz2 is 62.02 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.

So far nothing bad seems to happen, however we probably should use the Git Large File Storage for compressed tarballs?

UPDATE 0

We are looking into using Git Large File Storage for the compressed tarballs.

We will study how to use git lfs migrate as well.

We do NOT know if this is a good idea or not.

I don't know either. It didn't seem to be a problem though, did it? Just a warning? Without knowing what the LFS does I can't answer but as long as it doesn't cause a problem the way we have it I don't know that it's NECESSARY to enable?

xexyl commented 8 months ago

The results of test of make update may be found in commit 8aefcaa

As it was sometimes said over the radio in the past: "This is only a test ... beeeeeeeeeeeep!" :-)

The test was successful under macOS.

NOTE: The make update action should only be done by IOCCC judges.

Noted. Thanks.

xexyl commented 8 months ago

As per comment 2010767087, this TODO item has been marked complete:

  • Verify that the format of markdown files is good

But is it? We can't know until we look at the html files, right?

xexyl commented 8 months ago

As for the priorities comment I will have to look at this another time and probably not today. It's a lot of good content though and I'm sure I'll have questions before I can properly look at the issue (or at least part of it) but we shall see.

I have other things I'll be doing today and maybe the next few days but I'll be able to do more soon anyway and I did leave a comment that will necessitate me doing something, the issue of 2020/ferguson2/chocolate-cake files. But that's for another issue.

Have a great day!

lcn2 commented 8 months ago

When pushing commit 8aefcaa where the top level ioccc.tar.bz2 everything comprised tarball is formed, GitHub issues the following warnings:

remote: warning: See https://gh.io/lfs for more information.
remote: warning: File ioccc.tar.bz2 is 62.02 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.

So far nothing bad seems to happen, however we probably should use the Git Large File Storage for compressed tarballs?

UPDATE 0

We are looking into using Git Large File Storage for the compressed tarballs. We will study how to use git lfs migrate as well. We do NOT know if this is a good idea or not.

I don't know either. It didn't seem to be a problem though, did it? Just a warning? Without knowing what the LFS does I can't answer but as long as it doesn't cause a problem the way we have it I don't know that it's NECESSARY to enable?

We do not know the answers to your questions. We might have to experiment on a different repo to see what happens.

xexyl commented 8 months ago

When pushing commit 8aefcaa where the top level ioccc.tar.bz2 everything comprised tarball is formed, GitHub issues the following warnings:

remote: warning: See https://gh.io/lfs for more information.
remote: warning: File ioccc.tar.bz2 is 62.02 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.

So far nothing bad seems to happen, however we probably should use the Git Large File Storage for compressed tarballs?

UPDATE 0

We are looking into using Git Large File Storage for the compressed tarballs. We will study how to use git lfs migrate as well. We do NOT know if this is a good idea or not.

I don't know either. It didn't seem to be a problem though, did it? Just a warning? Without knowing what the LFS does I can't answer but as long as it doesn't cause a problem the way we have it I don't know that it's NECESSARY to enable?

We do not know the answers to your questions. We might have to experiment on a different repo to see what happens.

That sounds like a good idea. I will have to look back another time today or another day. I can barely see right now. Hope you have a great rest of your day!

lcn2 commented 8 months ago

As per comment 2010767087, this TODO item has been marked complete:

  • Verify that the format of markdown files is good

But is it? We can't know until we look at the html files, right?

We used scripts to verify the levels and form of headlines, plus visual inspection of the year and top level files.

If you think there are other problems, feel to advise.

UDPATE 0

.. and correct.

xexyl commented 8 months ago

As per comment 2010767087, this TODO item has been marked complete:

  • Verify that the format of markdown files is good

But is it? We can't know until we look at the html files, right?

We used scripts to verify the levels and form of headlines, plus visual inspection of the year and top level files.

If you think there are other problems, feel to advise.

UDPATE 0

.. and correct.

Oh I was just saying in case. I'm sure it will be fine or mostly fine (it after all can never be perfect nor should we try and make it perfect). If you looked at it that's good and maybe not much has to be done then. Perhaps nothing has to be done!

Well I'll get back sometime soon as noted in another thread (and above I think).

lcn2 commented 8 months ago

This just a note to ay that there is a bug in the way that Linux builds the compressed tarballs.

Now under Linux, make form_entry_tarball modifies all tarballs even though they are up to date.

Now under Linux, make form_year_tarball and form_all_tarball generates sed(1) errors.

We will work on fixing these.

UPDATE 0

Using new .tar.tstamp files, we have setup a way that both macOS and Linux can produce the same compressed tarballs.

The method used for macOS needs to be ported to Linux next: right now it does NOT work under Linux.

UPDATE 1

First we need to change the method by which bin/tar-entry.sh determines if the entry's compressed tarball needs to be replaced. We will start coding this method tomorrow.

lcn2 commented 8 months ago

With commit e9c3ec7ce703e32312efcc16a1282a38f943f5e1 the bin/tar-entry.sh tool has been greatly improved. Compressed tarballs under macOS via make form_entry_tarball when fetched under Linux will not be updated when make form_entry_tarball is run on the same files.

While the test of the bin/tar-entry.sh tool worked well under Linux.

However, before we get to the above TODOs, we need tor resolve a problem under Linux for make verify_entry_files:

$ make verify_entry_files
...
bin/all-run.sh: debug[3]: starting to process year/dir: 2018/burton2
bin/chk-entry.sh: Warning: file list does NOT match manifest for: 2018/burton2
bin/chk-entry.sh: Warning: manifest files that are missing starts below: 2018/burton2
2018/burton2/c++11
bin/chk-entry.sh: Warning: manifest files that are missing ends above: 2018/burton2
bin/chk-entry.sh: Warning: found files not in manifest starts below: 2018/burton2
2018/burton2/c11
bin/chk-entry.sh: Warning: found files not in manifest ends above: 2018/burton2
bin/all-run.sh: ERROR: tool: bin/chk-entry.sh  -- 2018/burton2 failed, error: 1
bin/all-run.sh: Warning: EXIT_CODE set to: 1
...

UPDATE 0

In particular under macOS this command is OK but fails under Linux:

bin/chk-entry.sh -v 3 2018/burton2

We will try to this issue when we return in about 2 days.

xexyl commented 8 months ago

Good comments above. I'm sorry I'm only seeing them. I am not sure they all need a reply though. I'm working on the manifest update now. I did notice some issues with 2020/ferguson2 but that'll happen when I work on the manifest more generally. What I am working on is your suggestions with 2020/ferguson2. I'm unsure if I'll be doing any more after that today. Tomorrow I have an appointment with the audiologist but I might be able to actually get into things tomorrow. I hope so.

Are there any comments above that would be good for me to read thoroughly and or reply to? Let me rephrase that: the above comments from the past few days. Well I will read them more thoroughly next time I look back here but I mean is there anything that needs to be replied to? Perhaps the one that does is the priorities one but the other two, the ones about the linux issue with the tar scripts. Do they need any comments? Looking at them now it seems not.

xexyl commented 8 months ago

Comments, Suggestions as Questions welcome

Any Comments, Suggestions as Questions you may have about this comment are welcome.

I invariably will have comments and questions and possibly suggestions too. I noticed some things (as noted above) but I will check the other entries as well. I agree that some issues are connected so it would be good to work on those together.

However I only briefly looked at the comment I am replying to so I'll have more to say when I return to this: probably not today but maybe tomorrow (I hope for tomorrow).

xexyl commented 8 months ago

I have several commits incoming .. just doing make quick_www to make sure all is good. if so I'll do the commit for that and then I will open a pull request. Then I am afraid indeed that I will be doing other things for the rest of the day. I'm just happy I got as much as I did today given how hard last night was.

lcn2 commented 8 months ago

Are there any comments above that would be good for me to read thoroughly and or reply to? Let me rephrase that: the above comments from the past few days. Well I will read them more thoroughly next time I look back here but I mean is there anything that needs to be replied to? Perhaps the one that does is the priorities one but the other two, the ones about the linux issue with the tar scripts. Do they need any comments? Looking at them now it seems not.

comment 2010746084, as revised, is import as to the top priority work on two ((top priority)) issues.

lcn2 commented 8 months ago

The Linux problem reported in comment 2019517975 has been fixed.

So next (perhaps starting the day after tomorrow):

lcn2 commented 8 months ago

With commit f07fec3a98152fd1bcf338435c46bec5e0df07d4:

We eliminated the need for .tar.tstamp files, changed bin/tar-year.sh to use the new procedure from bin/tar-entry.sh that determines of a compressed tarball needs to be rebuilt.

TODO: Test the bin/tar-year.sh tool under Linux.

TODO: Address issues with building the top level ioccc.tar.bz2 nand the bin/tar-all.sh tool and the make form_all_tarball rule.

NOTE: The bin/tar-all.sh tool and the make form_all_tarball rule and the make tar should NOT be used at this time.

lcn2 commented 8 months ago

The top level ioccc.tar.bz2 compressed tarball is a number of problems:

Potential solutions:

  1. Use the Git Large File Storage (LFS) to manage the top level ioccc.tar.bz2 compressed tarball
  2. Release mulit-year compress tarball sets: group by decade or group by sets of every N-contents into several top level compressed tarballs
  3. Only release year level and entry level compressed tarballs

We are not sure which is better and so we will think about this for a day or few days. Comments welcome.

xexyl commented 8 months ago

The top level ioccc.tar.bz2 compressed tarball is a number of problems:

  • The size of the top level ioccc.tar.bz2 compressed tarball (> 50 Mbytes) is an issue with GitHub

  • Multiple updates to the top level ioccc.tar.bz2 compressed tarball adds considerable weight to the .git directory to hold all entries.

  • Every time anything is added, modified or removed from an entry the top level ioccc.tar.bz2 compressed tarball has to be updated.

Potential solutions:

  1. Use the Git Large File Storage (LFS) to manage the top level ioccc.tar.bz2 compressed tarball

  2. Release mulit-year compress tarball sets: group by decade or group by sets of every N-contents into several top level compressed tarballs

  3. Only release year level and entry level compressed tarballs

We are not sure which is better and so we will think about this for a day or few days. Comments welcome.

I do indeed have some thoughts but I need sleep. Today was a long day.

I have a question about the manifest that should be answered before I make commits with it but I will have to ask that tomorrow (I hope).

I was at the audiologist getting hearing aids and that took some time and I had a pretty hard night on top of that.

I managed to read a little bit today but I could barely see so these are some of the reasons I didn't do anything here today. I guess tomorrow I will at least be able to reply to the comments that I have unfortunately mostly neglected the last some days (sorry about that!).

Once the question I have is answered I can make fixes and improvements to the manifest.

As far as the tarball issue I will also sleep on it.

Have a great night!

lcn2 commented 8 months ago

With commit e4a76879ba987d292bced75fdb42e6954b5c4de2 we have been successful in testing bin tools that form compressed tarballs for entries and IOCCC years and untar the same compressed tarballs. This worked well under both macOS (macOS 14.4.1) and Linux (RHEL9.3).

xexyl commented 8 months ago

With commit e4a7687 we have been successful in testing bin tools that form compressed tarballs for entries and IOCCC years and untar the same compressed tarballs. This worked well under both macOS (macOS 14.4.1) and Linux (RHEL9.3).

Does this address the problem you asked about yesterday? If not: I think it'd be good if we could have an ioccc.tar.bz2 file that is the entire contest history but if this is not possible with say, LFS, then it might be good to have both the YYYY.tar.bz2 and also maybe a decade. I think it'd be good to have a YYYY.tar.bz2 for each year but to have more than one year is ultimately a good thing if possible.

xexyl commented 8 months ago

Also how did you find out that sed -i '' is not portable? If so what versions does it not work? I ask of course because if there can be a workaround in sgit(1) that would be good though I have tested it with macOS sed and also GNU sed, so...

Thanks.

xexyl commented 8 months ago

Comments, Suggestions as Questions welcome

Any Comments, Suggestions as Questions you may have about this comment are welcome.

I have one that will be important before I start making changes. Important in case you have to make changes to the manifest as I don't want to have to figure out what I added and how to merge a Numbers spreadsheet.

Previously there was a three part commit for new files or file name changes. How do you wish commits to be processed (which commits, what stages etc.) for fixing the manifest in text and content of the spreadsheet? That I deem is important before I can really start working on it. That being said I can hopefully look at the index.html pages and determine if there are problems like you suggested (or others I notice) that can be fixed at the same time as the manifest.

As for those issues, the index.html and other things in the comment I am replying to (but snipped out most), I will try replying to later today or maybe in just a bit. That is not certain but once the manifest commit questions are answered I absolutely have fixes that I can make. It might even be okay to just go through the entire spreadsheet and fix all those things to make it 'clean', say, and then I can make adjustments as I look at each of the over 300 index.html pages. That I feel might be easier for me to deal with. Why? Because then I don't have to go back and forth to Safari and the console. But that will be determined later.

I do need to find the GitHub rendered html pages though ... I'm sure that's in one of the issues here.

xexyl commented 8 months ago

QUESTION for make www, make quick_www and related rules

Since it takes quite some time to run each time even if one file only is modified, is it a problem that you can think of to do:

make YEARS="2020 2019" quick_www

for example ?

Of course it'll still run the other rules that don't use the YEARS variable but it would speed things up a fair bit. On that note is it a problem if the YYYY/Makefiles have those specific rules (the ones that act on the YYYY)? I can look at adding this if you think it's worth it. I mean it might not be useful later on but right now it might be.

xexyl commented 8 months ago

QUESTION for make www, make quick_www and related rules

Since it takes quite some time to run each time even if one file only is modified, is it a problem that you can think of to do:

make YEARS="2020 2019" quick_www

for example ?

Of course it'll still run the other rules that don't use the YEARS variable but it would speed things up a fair bit. On that note is it a problem if the YYYY/Makefiles have those specific rules (the ones that act on the YYYY)? I can look at adding this if you think it's worth it. I mean it might not be useful later on but right now it might be.

Nope .. definitely not okay to redefine YEARS. I just noticed a problem with that. So the question is if the YYYY/Makefiles can have rules that only act on the html files.

Would that be possible?

... the fact I tested it after running it without redefining YEARS means I have to do it all over, unfortunately, but once I do that I have a commit as I found a space at an end of a line in a markdown file, one I added the other day by accident.

xexyl commented 8 months ago

BTW: a tip for not adding spaces at the end of markdown files in vim. Add in your .vimrc the following:

autocmd! BufWritePre *.md %s/\s\+$//e

That will remove spaces of *.md files at the end of lines when you save the file. You can of course specify other globs as well.