Closed lcn2 closed 1 month ago
What if workdir is a topdir/.workdir
? It should be ignored right, but still usable for preparing the submission.
Also you might want to remind/warn people about specifying workdir
as /tmp
or under /tmp
, which some systems wipe on reboot, but seems like the natural place for workdir
to do work. Would not have been better from a UI/UX point of view to avoid the need to specify a workdir
and just always use /var/tmp
, which is not normally cleared on reboot and a natural place for work.
Failing the idea of using /var/tmp
. Maybe use a default like $HOME/.mkiocccentry
and again save the need to provide a working directory some place.
Also version string handling would probably be easier if the date was not part of the version. Seems redundant. Several package managers, like npm
, support semver notations for specifying version ranges to include, exclude dependencies. Maybe such notation could be used to resolve what versions a contest supports; again made easier if the dates were not part of the version string (but that is a small nit pick).
I do not recall the npm
notation supported, but sure that can be easily found and there are npm
packages that handle this, there is probably a C library already, else take a JavaScript version and port it to C (don't try to reinvent this).
Also version string handling would probably be easier if the date was not part of the version. Seems redundant. Several package managers, like
npm
, support semver notations for specifying version ranges to include, exclude dependencies. Maybe such notation could be used to resolve what versions a contest supports; again made easier if the dates were not part of the version string (but that is a small nit pick).I do not recall the
npm
notation supported, but sure that can be easily found and there arenpm
packages that handle this, there is probably a C library already, else take a JavaScript version and port it to C (don't try to reinvent this).
RPMs typically don't have the date in the version but they CAN.
I have my own RPM repository.
And it's not hard with the date anyway as we have our ways. It's actually not redundant because it allows us to know when it was modified. It's easier to keep track of.
One can always use a quick pipeline to get just the version number to use with say verge like the website scripts do.
What do you find problematic about it? If you did say just say that and I will read again when I have more time. But wanted to reply quickly.
Will not return until tomorrow.
Thanks for the feedback either way! I am genuinely wondering; we didn't see a problem but maybe you have something else?
Maybe that was npm? Not sure now. I will read with my reading glasses tomorrow. Distance glasses are too problematic.
Have a good night!
What if workdir is a
topdir/.workdir
? It should be ignored right, but still usable for preparing the submission.
Such "dot-dies" and "dot-files" are ignored by mkiocccentry(1)
so they will not be included in the formation of the submission's compressed tarball.
If one were to try and "sneak" then in, say by adding them to the compressed tarball, then txzchk(1)
and chkentry(1)
would flag them as errors.
TL;DR
If it OK to keep things like .git
, .github
, .exrc
, .vimrc
under your topdir
tree as mkiocccentry(1)
will ignore them.
Also you might want to remind/warn people about specifying
workdir
as/tmp
or under/tmp
, which some systems wipe on reboot, but seems like the natural place forworkdir
to do work. Would not have been better from a UI/UX point of view to avoid the need to specify aworkdir
and just always use/var/tmp
, which is not normally cleared on reboot and a natural place for work.
There is nothing wrong with having workdir
under /tmp
or under /tmp
. Once one as picked up the submission compressed tarball and uploaded it to the submit server, the contents of workdir
is no longer needed.
There is NO need to warm. In fact, using workdir
under /tmp
or under /tmp
is a good idea!
Well I had no idea of the utility of keeping a workdir
once a submission had be uploaded. I recall glimpses of comments going on how workdir
is special. If it life is short lived, then why do we need to create a workdir
ourselves?
In fact, using workdir [as] /tmp or under /tmp is a good idea!
I do have them some times. Good ideas. The whole workdir thing seemed over done. Just remembered, you probably want to use the user's preferred environment variable $TMPDIR
when defined otherwise /tmp
.
The stat(2)
on both the workdir
and topdir
args and keep their device (.st_dev
) and inode number (.st_inode
) values (in a global if that is easier).
Then as you was walking topdir
, flag as something for mkiocccentry(1)
to ignore if the node you find has the same device (.st_dev
) and inode number (.st_inode
) values as those you saved for workdir
. This will catch even the same where someone tried to "hard link" their workdir
under topdir
even if show up as a different name.
If you need (we forget if you do) walk workdir
, then flag as an error if the node you find has the same device (.st_dev
) and inode number (.st_inode
) values as those you saved for topdir
.
An equivalent firewall would apply to workdir
if you meed to walk workdir
too.
If course, if the node that has the same same device (.st_dev
) and inode number (.st_inode
) values is something to be ignored, then there is no error to flag.
This would be OK:
$ mkiocccentry topdir/.workdir topdir
because the topdir/.workdir
is a "dot-dir".
In light of GH-issuecomment-2705378721, do not flag finding workdir
(via a device (.st_dev
) and inode number (.st_inode
) value match) as an error, treat it as something to ignore.
directory loop insanity
The Guatemalan Insanity Pepper The Directory Loop Insanity Gummy
Well I had no idea of the utility of keeping a
workdir
once a submission had be uploaded. I recall glimpses of comments going on howworkdir
is special. If it life is short lived, then why do we need to create aworkdir
ourselves?In fact, using workdir [as] /tmp or under /tmp is a good idea!
I do have them some times. Good ideas. The whole workdir thing seemed over done. Just remembered, you probably want to use the user's preferred environment variable
$TMPDIR
when defined otherwise/tmp
.
More than one slot can by put under the same workdir
. If you look at a multi-slot mkioccentry
run, you will file the multiple workdir/UUID-slot_num/
directories showing you the contents that were used to package your submission workdir/UUID-slot_num.timestamp.txz
compressed tarball.
The existence of workdir
is NOT a problem for a subsequent mkiocccentry(1)
run, it is the workdir/UUID-slot_num/
directories that would be in conflict.
So consider this:
$ cd mkioccentry
$ cd test_ioccc/slot/good
$ for X in $(seq 0 9); do echo =-=-= starting $X =-=-=; echo; ; mkiocccentry -i answers/12345678-1234-4321-abcd-1234567890ab-$X -Y /tmp topdir/12345678-1234-4321-abcd-1234567890ab-$X; done
It is the /tmp/12345678-1234-4321-abcd-1234567890ab-X/
trees that need to be removed if you want to re-run that for loop:
$ rm -rf /tmp/12345678-1234-4321-abcd-1234567890ab-?
$ for X in $(seq 0 9); do echo =-=-= starting $X =-=-=; echo; ; mkiocccentry -i answers/12345678-1234-4321-abcd-1234567890ab-$X -Y /tmp topdir/12345678-1234-4321-abcd-1234567890ab-$X; done
Well in that case create $HOME/.mkiocccentry
as I menetioned earlier.
it is the workdir/UUID-slot_num/ directories that would be in conflict.
Which again, why not just always overwrite them if they're disposable and forego need for a -f
force option. I don't understand why they can't be overwritten with each run? I either want to create an initial submission, need to restart an interrupted mkiocccentry run, or need to update a submission all of which need to replace the UUID-SLOT
directory. Why would I want to keep past version of a submission?
Well I had no idea of the utility of keeping a
workdir
once a submission had be uploaded. I recall glimpses of comments going on howworkdir
is special. If it life is short lived, then why do we need to create aworkdir
ourselves?In fact, using workdir [as] /tmp or under /tmp is a good idea!
I do have them some times. Good ideas. The whole workdir thing seemed over done. Just remembered, you probably want to use the user's preferred environment variable
$TMPDIR
when defined otherwise/tmp
.More than one slot can by put under the same
workdir
. If you look at a multi-slotmkioccentry
run, you will file the multipleworkdir/UUID-slot_num/
directories showing you the contents that were used to package your submissionworkdir/UUID-slot_num.timestamp.txz
compressed tarball.The existence of
workdir
is NOT a problem for a subsequentmkiocccentry(1)
run, it is theworkdir/UUID-slot_num/
directories that would be in conflict.So consider this:
$ cd mkioccentry $ cd test_ioccc/slot/good $ for X in $(seq 0 9); do echo =-=-= starting $X =-=-=; echo; ; mkiocccentry -i answers/12345678-1234-4321-abcd-1234567890ab-$X -Y /tmp topdir/12345678-1234-4321-abcd-1234567890ab-$X; done
It is the
/tmp/12345678-1234-4321-abcd-1234567890ab-X/
trees that need to be removed if you want to re-run that for loop:$ rm -rf /tmp/12345678-1234-4321-abcd-1234567890ab-? $ for X in $(seq 0 9); do echo =-=-= starting $X =-=-=; echo; ; mkiocccentry -i answers/12345678-1234-4321-abcd-1234567890ab-$X -Y /tmp topdir/12345678-1234-4321-abcd-1234567890ab-$X; done
Yup. One can use the same workdir if they choose. One can also have it be a subdirectory in their submission directory. But this should be up to each person.
For the test-suite we use the same workdir several times one after another.
And as far as environmental variables. That can be a hairy thing.
It can be done safely but it especially should not be done if we're going to perform potentially dangerous commands (or functionality). And that even includes simply copying files.
As far as the issue itself.
I don't know how to go about it.
I mean does it mean that if it encounters that directory it should use fts_set() to not traverse it?
As far as invalid version it seems like that might be not the most recent version installed? I didn't look in detail.
I need to get some sleep. Waking up too early. But tomorrow I have a call and unfortunately it's often late.
Nonetheless I can resolve at least some of the opened issues before that.
Feel free to assign this to me and say what you think might be done. As I wrote the fts stuff (including the much enhanced interface) I know this part quite well.
Better go now. Good night! Let me know what you wish.
And yes as far as version strings go there's also the checks by chkentry indeed so even of we were to change it it could not be done during this contest.
If we do it it also means that other repos will also have to be updated. I don't think it's worth it and I think it's actually good to have the date as it is easier to track when something was committed.
Suggestion
The
stat(2)
on both theworkdir
andtopdir
args and keep their device (.st_dev
) and inode number (.st_inode
) values (in a global if that is easier).Then as you was walking
topdir
, flag as an error if the node you find has the same device (.st_dev
) and inode number (.st_inode
) values as those you saved forworkdir
. This will catch even the same where someone tried to "hard link" theirworkdir
undertopdir
even if show up as a different name.If you need (we forget if you do) walk
workdir
, then flag as an error if the node you find has the same device (.st_dev
) and inode number (.st_inode
) values as those you saved fortopdir
.As a firewall, and only if this is easy, while you are walking
topdir
AND you are NOT at the "root oftopdir
tree), flag as an error if the node you find has the same device (.st_dev
) and inode number (.st_inode
) values as those you saved fortopdir
. This would indicate a directory loop insanity cycle. That shouldn't happen, but you might as well add this firewall (or paranoia) check if you are already looking for theworkdir
values.An equivalent firewall would apply to
workdir
if you meed to walkworkdir
too.
As far as cycle in directory tree. Isn't the check for FTS_DC it?
And as far as stat goes: I make sure stat(2) info is available.
Anyway we read it properly tomorrow.
If you feel I have said enough yet to reply to please do; otherwise I will get a more thorough reply tomorrow.
Really must get some sleep. Good night!
I mean does it mean that if it encounters that directory it should use fts_set() to not traverse it?
Yes. Good point.
So instead of marking it as an "error", treat it as something to ignore. And then by ignoring it, one would not traverse it as well.
The suggestion comment has been adjusted accordingly.
As far as cycle in directory tree. Isn't the check for FTS_DC it?
Yes, it does. So this is not a problem.
The suggestion comment has been adjusted accordingly.
And as far as stat goes: I make sure stat(2) info is available.
Sure. You probably do not need to make separate calls to stat(2)
if fts_set()
and friends provide you with the device and inode number already.
Is there an existing issue for this?
- [x] I have searched for existing issues and did not find anything like this
Describe the bug
Some people place the
workdir
undertopdir
.We found an un-tarred tree with
topdir/workdir/UUID/
where UUID was the username.
That's odd but your idea is good .. I can do this per discussion below. I won't update the version for obvious reasons. It would indeed complicate matters. On the other hand if we were to use verge (as executed) we could check version differences. Problem with that is that make install
does not do that by default. Now the install server might but that's another matter entirely. I'll get back to that below.
The
topdir/workdir/UUID/
directory is empty.The
topdir/workdir/' directory only has the
UUID` directory under it.What you expect
The
mkiocccentry(1)
tool would disallow aworkdir
anwyere undertopdir
.
Right now I'm trying to parse this in my tired head. I think when I see below it'll make more sense - and if not when I'm actually more awake I will manage fine.
Environment
- OS: unknown
- Device: unknown
- Compiler: unknown
FYI, the
.info.json
showed:... "IOCCC_info_version" : "1.16 2024-05-18", ... "mkiocccentry_version" : "2.0.0 2025-02-28", "iocccsize_version" : "28.15 2024-06-27", "chkentry_version" : "2.0.0 2025-02-28", "fnamchk_version" : "2.0.0 2025-02-28", "txzchk_version" : "2.0.0 2025-02-28",
bug_report.sh output
n/a
Anything else?
When we run
chkentry(1)
on the un-tarred tree we see these errors:... .auth.json validation error ERROR[149]: chk_mkiocccentry_version: invalid mkiocccentry_version .auth.json validation error ERROR[120]: chk_chkentry_version: invalid chkentry_version ... .info.json validation error ERROR[149]: chk_mkiocccentry_version: invalid mkiocccentry_version .info.json validation error ERROR[120]: chk_chkentry_version: invalid chkentry_version .info.json validation error ERROR[172]: chk_txzchk_version: invalid txzchk_version ... ERROR[1]: main: one or more tests failed
So it is also possible that someone has an outdated mkiocccentry toolkit.
That seems the case yes. But this check you discovered was never thought of. So although it would be invalid for other reasons this issue is still something to consider. Given that the directory was empty it would seem like it might be good to fix. We can put in news about this and you could put something on mastodon too. You might mention that people check the status of their submissions so they know if they have to fix something.
It is also worth noting that the
workdir
did NOT go into the.info.json
manifest because where were no files under theworkdir
tree that was put undertopdior
.
I actually thought of this when you said it was empty.
Important NOTE
IF this results in a change to
mkiocccentry(1)
(and that is a big IF), resulting in version changes to ANY if the version values found in.info.json
, then this WILL COMPLICATE the version checking.
I think it's not worth updating the version because as you say it would invalidate the version. I reread this now - a moment ago I was thinking you were wondering about code changing, not version changing. Yes not changing version but actually changing code is important indeed.
Consider, just as an example, that for some reason the
mkiocccentry_version
changes from "2.0.1 2025-03-02" to "2.0.2 2025-03-07". Remember, this is just as a hypothetical example!Then in order to NOT invalidate everyone else's submission, the version testing code CANNOT to a simple pattern match. Code such as that in
chkentry(1)
, for example, need to allow BOTH "2.0.1 2025-03-02" and "2.0.2 2025-03-07"!
Yup and this is ugly at the very least.
QUESTIONs
Q: If this a bug that needs to be fixed in the short term?
A: Probably not. This could be addressed as feature for IOCCC29.
I don't know. On the one hand it is a shame if people submitted these. But on the other hand it seems likely that they didn't run the tool correctly because if they had txzchk would have flagged the file missing!
Q: Do we simply update news to say "do not put workdir under topdir"? And if so, would that do any good if people don't notice the news asking people to not do this?
A: ???
Good question. It might be worth putting in the news AND on mastodon AND (if it works this way - don't know) on discord - make sure people check their submission status! You can also put in the note here.
Q: As far as the IOCCC judges are concerned, we can simply ignore any
workdir
that we find. Given that, it is a bug that needs to be fixed in the short term?A: Probably not. This could be addressed as feature for IOCCC29.
On the other hand ... it seems like I did not think of something pretty important. I did think of a lot of scenarios but as we all know we can easily miss something even when one does not need to get a lot done in a short time: but I had a great deal to get done in a short time!
Q: Is there a similar problem if where to "do it the other way around" and someone were to put
topdir
underworkdir
?A: ???
My guess is yes. So as you suggest I need to check the stat(2) info on both and make sure that if these things happen it is an ERROR.
I won't update the versions.
I mean does it mean that if it encounters that directory it should use fts_set() to not traverse it?
Yes. Good point.
So instead of marking it as an "error", treat it as something to ignore. And then by ignoring it, one would not traverse it as well.
Good my idea will work. I've done this in a number of places already.
The suggestion comment has been adjusted accordingly.
As far as cycle in directory tree. Isn't the check for FTS_DC it?
Yes, it does. So this is not a problem.
I thought so. The lower level util function read_fts()
handles this one.
The suggestion comment has been adjusted accordingly.
And as far as stat goes: I make sure stat(2) info is available.
Sure. You probably do not need to make separate calls to
stat(2)
iffts_set()
and friends provide you with the device and inode number already.
Well that's per directory entry in the tree so I still have to get stat(2)
info first - to compare. I have done this part already. Now to compare. I'll see if I can work out various conditions to test the new set up. It might be nice if I could come up with problematic ones first and then see if I can prevent those with the updates. I'll do that before I proceed.
Oh yes good catch @lcn2! I just tried this...
mkiocccentry foo_src/test/ foo_src/
and the submissions directory is a mess!
drwxr-xr-x 501/20 0 2025-03-07 02:54 test-0/
drwxr-xr-x 501/20 0 2025-03-07 02:54 test-0/test/
-r--r--r-- 501/20 14 2025-03-07 02:54 test-0/Makefile
drwxr-xr-x 501/20 0 2025-03-07 02:54 test-0/zab/
-r-xr-xr-x 501/20 23 2025-03-07 02:54 test-0/try.sh
-r--r--r-- 501/20 1106 2025-03-07 02:54 test-0/.auth.json
drwxr-xr-x 501/20 0 2025-03-07 02:54 test-0/foo/
-r--r--r-- 501/20 37 2025-03-07 02:54 test-0/prog.c
-r--r--r-- 501/20 8 2025-03-07 02:54 test-0/remarks.md.bak
-r--r--r-- 501/20 4 2025-03-07 02:54 test-0/remarks.md
drwxr-xr-x 501/20 0 2025-03-07 02:54 test-0/bar/
-r--r--r-- 501/20 1897 2025-03-07 02:54 test-0/.info.json
-r--r--r-- 501/20 0 2025-03-07 02:54 test-0/bar/foo
drwxr-xr-x 501/20 0 2025-03-07 02:54 test-0/bar/bar/
-r--r--r-- 501/20 5 2025-03-07 02:54 test-0/foo/foo
drwxr-xr-x 501/20 0 2025-03-07 02:54 test-0/foo/tmp/
-r--r--r-- 501/20 0 2025-03-07 02:54 test-0/zab/foo
drwxr-xr-x 501/20 0 2025-03-07 02:54 test-0/test/test-0/
Well at least I can easily test this ....
Using the same directory for work and topdir is also fun .... that has to be flagged as an error obviously. Kind of embarrassed I did not think of that one! Well I mean technically it's not an error: all tests pass. But still kind of a mess.
Same directory for work and top is now an error. Now to look at testing the others ...
Added extra sanity checks to the FTS code to jparse. Will sync to mkiocccentry next and then continue here.
Okay now if workdir is found under topdir it will not descend into it.
I have to do the check again in the check function too however. Once that's good I'll see what other conditions I might be able to find.
I think I'm done .. checking if I missed anything you thought of or encountered.
Updating man page for the -u
option (after doing some final tests there) and then will commit. This should resolve this problem as well as add the missing directories to skip.
I will also check the other issues in the repo that were opened. I won't address the force delete submission directory however as much as I'd like to get that out of the way (ha).
-u uuid
option works well. If it's invalid it'll just ask for a valid one. Will make it easier for people now. Checking other issues first.
Unless I am very much mistaken commit https://github.com/ioccc-src/mkiocccentry/pull/1211/commits/7c1bc2ea72d46d617e04e823736336616739650e should resolve this issue.
Please let me know @lcn2.
In regards to the Important note mentioned in the top comment of this issue:
See issue #1215
=
For this emergency fix example, we might need to change things in
soup/version.h
to:define
MKIOCCCENTRY_VERSION
"2.0.2 2025-04-01"define MIN_MKIOCCCENTRY_VERSION "2.0.1 2025-03-02"
This minimum allowed version capability might be needed sooner than IOCCC29 if we find ourselves in an emergency 🚨 fix situation while IOCCC28 is open.
Perhaps this comment should become its own issue?
Yes.
And for IOCCC29, right ?
Okay I just read your comment more carefully (I do have to leave very soon).
There is a thought I had that is not ideal but it might be okay.
We might integrate the version check code in verge and move it into util.
This is not something I particularly like but maybe we could make it more general.
I don't remember exactly how it works. We do have a jstr_util too. That's primarily for jstrdecode and jstrencode but it could be (I guess) put there: it is a string after all.
And it actually might be nice to split the util functions into multiple files anyway as it seems not ideal to have so many different kinds of utility functions in there.
A lot don't even fit in jparse but for historical reasons they of course are there.
It is unfortunate but it's how it can work for mkiocccentry.
Must leave! Hope to be back soon.
Oh! That's silly. We can just include verge.h and make sure it's linked in.
We believe we have addressed all of the current questions that still need answering at this time. If we've missed something or something else needs to be clarified, please ask again.
We realize that there may be some open questions, however issue #1215 was just created and so any important unanswered questions should be asked again in light of that matter.
We are going back to sleep for a bit.
We believe we have addressed all of the current questions that still need answering at this time. If we've missed something or something else needs to be clarified, please ask again.
We realize that there may be some open questions, however issue #1215 was just created and so any important unanswered questions should be asked again in light of that matter.
We are going back to sleep for a bit.
Sleep well!
Based on commit 768ed6aacd9e806a92ff30e1c81f005fbd23a088 this issue has been resolved.
Thank you to all who helped with this issue.
Is there an existing issue for this?
Describe the bug
Some people place the
workdir
undertopdir
.We found an un-tarred tree with
topdir/workdir/UUID/
where UUID was the username.The
topdir/workdir/UUID/
directory is empty.The
topdir/workdir/' directory only has the
UUID` directory under it.What you expect
The
mkiocccentry(1)
tool would disallow aworkdir
anwyere undertopdir
.Environment
FYI, the
.info.json
showed:bug_report.sh output
n/a
Anything else?
When we run
chkentry(1)
on the un-tarred tree we see these errors:So it is also possible that someone has an outdated mkiocccentry toolkit.
It is also worth noting that the
workdir
did NOT go into the.info.json
manifest because where were no files under theworkdir
tree that was put undertopdior
.Important NOTE
IF this results in a change to
mkiocccentry(1)
(and that is a big IF), resulting in version changes to ANY if the version values found in.info.json
, then this WILL COMPLICATE the version checking.Consider, just as an example, that for some reason the
mkiocccentry_version
changes from "2.0.1 2025-03-02" to "2.0.2 2025-03-07". Remember, this is just as a hypothetical example!Then in order to NOT invalidate everyone else's submission, the version testing code CANNOT to a simple string match. Code such as that in
chkentry(1)
, for example, might neef to allow BOTH "2.0.1 2025-03-02" and "2.0.2 2025-04-01"!See issue #1215 .
QUESTIONs
Q: If this a bug that needs to be fixed in the short term?
A: Probably not. This could be addressed as feature for IOCCC29.
Q: Do we simply update news to say "do not put workdir under topdir"? And if so, would that do any good if people don't notice the news asking people to not do this?
A: ???
Q: As far as the IOCCC judges are concerned, we can simply ignore any
workdir
that we find. Given that, it is a bug that needs to be fixed in the short term?A: Probably not. This could be addressed as feature for IOCCC29.
Q: Is there a similar problem if where to "do it the other way around" and someone were to put
topdir
underworkdir
?A: ???