Closed musliner closed 2 years ago
musliner
commented
2 years ago we havent seen this much, in quite a few runs, so it seems non-reproducible and relatively rare, hence possible race condition, but yields pretty much catastrophic failure of trial.
StephenGss
commented
2 years ago It's very likely this is an I/O issue on the host machine. We avoid this very well in our evaluations by using a ram drive for our logging purposes. A ram drive is not necessary by any means, but our program is sensitive to I/O interruptions. It will always catch up, but it may catch those events late because of this.
musliner
commented
2 years ago the host was definitely more loaded than usual during this run, but ... 13 minutes is a long time for the pal-tournament to be still responding to CHECK_COST commands but apparently logging nothing (and apparently thus not "catching up" ?).... is there a way to get it to flush more frequently?
musliner
commented
2 years ago Thanks Stephen,
I responded in the issue- is there a way to get it to flush output buffer more frequently?
--D
From: StephenGss @.> Sent: Monday, November 15, 2021 12:11 PM To: StephenGss/PAL @.> Cc: musliner @.>; Author @.> Subject: Re: [StephenGss/PAL] missing game-over (Issue #40)
It's very likely this is an I/O issue on the host machine. We avoid this very well in our evaluations by using a ram drive for our logging purposes. A ram drive is not necessary by any means, but our program is sensitive to I/O interruptions. It will always catch up, but it may catch those events late because of this.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/StephenGss/PAL/issues/40#issuecomment-969181916 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AEKHIL3T333NUVEPANGUCBLUMFEMNANCNFSM5ICHOX3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .
StephenGss
commented
2 years ago Sorry, I missed the last sentence in your original message. I'll take a look at our eval tournament manager and compare it to the release 2.0 one. There may be some optimizations that will help here.
StephenGss
commented
2 years ago Does that log file include the debug log? I wonder if the tournament manager sent a RESET that didn't go through for some reason. This could be the case if the debug log ends with "Reset Triggered... "
musliner
commented
2 years ago I’m not sure what you mean by debug log… that log is from running
$thisdir/../prt/timestamp ./gradlew runClient –stacktrace
If we should be turning on more flags, just lemme know
Or if the game leaves some other log somewhere we should look at…
thx
From: StephenGss @.> Sent: Monday, November 15, 2021 12:55 PM To: StephenGss/PAL @.> Cc: musliner @.>; Author @.> Subject: Re: [StephenGss/PAL] missing game-over (Issue #40)
Does that log file include the debug log? I wonder if the tournament manager sent a RESET that didn't go through for some reason. This could be the case if the debug log ends with "Reset Triggered... "
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/StephenGss/PAL/issues/40#issuecomment-969215852 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AEKHIL6MZYXF4TZTTURBQK3UMFJQFANCNFSM5ICHOX3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .
musliner
commented
2 years ago I’m not sure what you mean by debug log… that log is from running $thisdir/../prt/timestamp ./gradlew runClient –stacktrace
If we should be turning on more flags, just lemme know
Or if the game leaves some other log somewhere we should look at…
thx
musliner
commented
2 years ago Are you using our launcher? There should be different logs in PolycraftAIGym/Logs/
From: David Musliner @.> Sent: Monday, November 15, 2021 1:37 PM To: 'StephenGss/PAL' @.>; 'StephenGss/PAL' @.> Cc: 'musliner' @.>; 'Author' @.>; Goss, Stephen Andrew @.> Subject: RE: [StephenGss/PAL] missing game-over (Issue #40)
I'm not sure what you mean by debug log... that log is from running $thisdir/../prt/timestamp ./gradlew runClient -stacktrace
If we should be turning on more flags, just lemme know
Or if the game leaves some other log somewhere we should look at...
thx
From: StephenGss @.**@.>> Sent: Monday, November 15, 2021 12:55 PM To: StephenGss/PAL @.**@.>> Cc: musliner @.**@.>>; Author @.**@.>> Subject: Re: [StephenGss/PAL] missing game-over (Issue #40)
Does that log file include the debug log? I wonder if the tournament manager sent a RESET that didn't go through for some reason. This could be the case if the debug log ends with "Reset Triggered... "
- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FStephenGss%2FPAL%2Fissues%2F40%23issuecomment-969215852&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C6aeaebc2e98d4ff8736908d9a86f5dfc%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726018549684100%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ReCRpArZNOrld7hoZM7eIl4%2FEWJlf2fswUT3Mvd239I%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAEKHIL6MZYXF4TZTTURBQK3UMFJQFANCNFSM5ICHOX3A&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C6aeaebc2e98d4ff8736908d9a86f5dfc%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726018549684100%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZwCKUhJcxxAWRm7JdJaFvzMe5Nm5NAa1i%2FjAMUMsTmk%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C6aeaebc2e98d4ff8736908d9a86f5dfc%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726018549694096%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SW%2Fl94iBwUGGICefkWHBFmV99r5UN6o33Df3ceyufWw%3D&reserved=0 or Androidhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C6aeaebc2e98d4ff8736908d9a86f5dfc%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726018549704087%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=312kh6ZR5aMDO8G9qiPqDK4ksbYZGdILeHmi8MS0jbk%3D&reserved=0.
musliner
commented
2 years ago We're still stuck on modified versions of old launcher.would be nice to update, but there hasn't been time .
--D
From: Goss, Stephen Andrew @.> Sent: Tuesday, November 16, 2021 9:33 AM To: David Musliner @.>; 'StephenGss/PAL' @.>; 'StephenGss/PAL' @.> Cc: 'musliner' @.>; 'Author' @.> Subject: RE: [StephenGss/PAL] missing game-over (Issue #40)
Are you using our launcher?
There should be different logs in PolycraftAIGym/Logs/
From: David Musliner @. @.> > Sent: Monday, November 15, 2021 1:37 PM To: 'StephenGss/PAL' @.>; 'StephenGss/PAL' @. @.> > Cc: 'musliner' @. @.> >; 'Author' @. @.> >; Goss, Stephen Andrew @. @.***> > Subject: RE: [StephenGss/PAL] missing game-over (Issue #40)
I'm not sure what you mean by debug log. that log is from running
$thisdir/../prt/timestamp ./gradlew runClient -stacktrace
If we should be turning on more flags, just lemme know
Or if the game leaves some other log somewhere we should look at.
thx
From: StephenGss @. @.>
Sent: Monday, November 15, 2021 12:55 PM To: StephenGss/PAL @. @.> > Cc: musliner @. @.> >; Author @. @.> > Subject: Re: [StephenGss/PAL] missing game-over (Issue #40)
Does that log file include the debug log? I wonder if the tournament manager sent a RESET that didn't go through for some reason. This could be the case if the debug log ends with "Reset Triggered... "
- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co m%2FStephenGss%2FPAL%2Fissues%2F40%23issuecomment-969215852&data=04%7C01%7CS tephen.Goss%40utdallas.edu%7C6aeaebc2e98d4ff8736908d9a86f5dfc%7C8d281d1d9c4d 4bf7b16e032d15de9f6c%7C0%7C0%7C637726018549684100%7CUnknown%7CTWFpbGZsb3d8ey JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdat a=ReCRpArZNOrld7hoZM7eIl4%2FEWJlf2fswUT3Mvd239I%3D&reserved=0 , or unsubscribe https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co m%2Fnotifications%2Funsubscribe-auth%2FAEKHIL6MZYXF4TZTTURBQK3UMFJQFANCNFSM5 ICHOX3A&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C6aeaebc2e98d4ff8736908d 9a86f5dfc%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726018549684100%7C Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC JXVCI6Mn0%3D%7C3000&sdata=ZwCKUhJcxxAWRm7JdJaFvzMe5Nm5NAa1i%2FjAMUMsTmk%3D&r eserved=0 . Triage notifications on the go with GitHub Mobile for iOS https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.appl e.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8% 26pt%3D524675&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C6aeaebc2e98d4ff87 36908d9a86f5dfc%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726018549694 096%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h aWwiLCJXVCI6Mn0%3D%7C3000&sdata=SW%2Fl94iBwUGGICefkWHBFmV99r5UN6o33Df3ceyufW w%3D&reserved=0 or Android.
musliner
commented
2 years ago I can relate to that lol. Can you point me to the launcher you use and maybe I can recommend some updates.
I think there were some significant changes we made to address those types of issues during the last evaluation.
From: David Musliner @.> Sent: Tuesday, November 16, 2021 10:21 AM To: Goss, Stephen Andrew @.>; 'StephenGss/PAL' @.>; 'StephenGss/PAL' @.> Cc: 'musliner' @.>; 'Author' @.> Subject: RE: [StephenGss/PAL] missing game-over (Issue #40)
We're still stuck on modified versions of old launcher...would be nice to update, but there hasn't been time ...
--D
From: Goss, Stephen Andrew @.**@.>> Sent: Tuesday, November 16, 2021 9:33 AM To: David Musliner @.**@.>>; 'StephenGss/PAL' @.**@.>>; 'StephenGss/PAL' @.**@.>> Cc: 'musliner' @.**@.>>; 'Author' @.**@.>> Subject: RE: [StephenGss/PAL] missing game-over (Issue #40)
Are you using our launcher? There should be different logs in PolycraftAIGym/Logs/
From: David Musliner @.**@.>> Sent: Monday, November 15, 2021 1:37 PM To: 'StephenGss/PAL' @.**@.>>; 'StephenGss/PAL' @.**@.>> Cc: 'musliner' @.**@.>>; 'Author' @.**@.>>; Goss, Stephen Andrew @.**@.>> Subject: RE: [StephenGss/PAL] missing game-over (Issue #40)
I'm not sure what you mean by debug log... that log is from running $thisdir/../prt/timestamp ./gradlew runClient -stacktrace
If we should be turning on more flags, just lemme know
Or if the game leaves some other log somewhere we should look at...
thx
From: StephenGss @.**@.>> Sent: Monday, November 15, 2021 12:55 PM To: StephenGss/PAL @.**@.>> Cc: musliner @.**@.>>; Author @.**@.>> Subject: Re: [StephenGss/PAL] missing game-over (Issue #40)
Does that log file include the debug log? I wonder if the tournament manager sent a RESET that didn't go through for some reason. This could be the case if the debug log ends with "Reset Triggered... "
- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FStephenGss%2FPAL%2Fissues%2F40%23issuecomment-969215852&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C4304b8d02f9947fb835d08d9a91d08b2%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726764430573528%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5n7nDuzAZl%2BNX2BIPGjWaTDwvU8UzNfwr1wFg3qrCtc%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAEKHIL6MZYXF4TZTTURBQK3UMFJQFANCNFSM5ICHOX3A&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C4304b8d02f9947fb835d08d9a91d08b2%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726764430583521%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=pzgSFFhTCKThk%2FkyuJmz1HZm4urO%2FdaFJ41%2FjyN0auU%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C4304b8d02f9947fb835d08d9a91d08b2%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726764430593514%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FS84gBTOI%2Beh%2FUeGlYheeUCu7QQZS1P9ZKU987kiQ0s%3D&reserved=0 or Androidhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C4304b8d02f9947fb835d08d9a91d08b2%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726764430593514%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=gZEqXHbFdMJpVHXh%2FIppoXAvWXB%2Fkghjtjo821Opc0A%3D&reserved=0.
musliner
commented
2 years ago In code/test/ there are a slew of LaunchTournament-v2*py scripts tailored to each test, alas. the main diffs being a list of json configs to run, and possibly MAX_TIME or somesuch.
If the i/o mess in those is responsible for some of the timing glitches etc, we should definitely tackle the upgrade. I had not looked in those for a long long time, but now I see that they may indeed be the source of the missed game ending (recently) b/c the game RESET is actually triggered in there (I totally didn't know that.)... yikes.
So yes, if you have update pointers or we just need to read some newer docs, we'll get on that.
--D
From: Goss, Stephen Andrew @.> Sent: Wednesday, November 17, 2021 12:18 AM To: David Musliner @.>; 'StephenGss/PAL' @.>; 'StephenGss/PAL' @.> Cc: 'musliner' @.>; 'Author' @.> Subject: RE: [StephenGss/PAL] missing game-over (Issue #40)
I can relate to that lol.
Can you point me to the launcher you use and maybe I can recommend some updates.
I think there were some significant changes we made to address those types of issues during the last evaluation.
From: David Musliner @. @.> > Sent: Tuesday, November 16, 2021 10:21 AM To: Goss, Stephen Andrew @. @.> >; 'StephenGss/PAL' @. @.>
; 'StephenGss/PAL' @. @.>
Cc: 'musliner' @. @.> >; 'Author' @. @.> > Subject: RE: [StephenGss/PAL] missing game-over (Issue #40)
We're still stuck on modified versions of old launcher.would be nice to update, but there hasn't been time .
--D
From: Goss, Stephen Andrew @. @.> > Sent: Tuesday, November 16, 2021 9:33 AM To: David Musliner @. @.> >; 'StephenGss/PAL' @. @.>
; 'StephenGss/PAL' @. @.>
Cc: 'musliner' @. @.> >; 'Author' @. @.> > Subject: RE: [StephenGss/PAL] missing game-over (Issue #40)
Are you using our launcher?
There should be different logs in PolycraftAIGym/Logs/
From: David Musliner @. @.> > Sent: Monday, November 15, 2021 1:37 PM To: 'StephenGss/PAL' @. @.>
; 'StephenGss/PAL' @. @.>
Cc: 'musliner' @. @.> >; 'Author' @. @.> >; Goss, Stephen Andrew @. @.> > Subject: RE: [StephenGss/PAL] missing game-over (Issue #40)
I'm not sure what you mean by debug log. that log is from running
$thisdir/../prt/timestamp ./gradlew runClient -stacktrace
If we should be turning on more flags, just lemme know
Or if the game leaves some other log somewhere we should look at.
thx
From: StephenGss @. @.>
Sent: Monday, November 15, 2021 12:55 PM To: StephenGss/PAL @. @.> > Cc: musliner @. @.> >; Author @. @.> > Subject: Re: [StephenGss/PAL] missing game-over (Issue #40)
Does that log file include the debug log? I wonder if the tournament manager sent a RESET that didn't go through for some reason. This could be the case if the debug log ends with "Reset Triggered... "
- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co m%2FStephenGss%2FPAL%2Fissues%2F40%23issuecomment-969215852&data=04%7C01%7CS tephen.Goss%40utdallas.edu%7C4304b8d02f9947fb835d08d9a91d08b2%7C8d281d1d9c4d 4bf7b16e032d15de9f6c%7C0%7C0%7C637726764430573528%7CUnknown%7CTWFpbGZsb3d8ey JWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdat a=5n7nDuzAZl%2BNX2BIPGjWaTDwvU8UzNfwr1wFg3qrCtc%3D&reserved=0 , or unsubscribe https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.co m%2Fnotifications%2Funsubscribe-auth%2FAEKHIL6MZYXF4TZTTURBQK3UMFJQFANCNFSM5 ICHOX3A&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C4304b8d02f9947fb835d08d 9a91d08b2%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726764430583521%7C Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC JXVCI6Mn0%3D%7C3000&sdata=pzgSFFhTCKThk%2FkyuJmz1HZm4urO%2FdaFJ41%2FjyN0auU% 3D&reserved=0 . Triage notifications on the go with GitHub Mobile for iOS https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.appl e.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8% 26pt%3D524675&data=04%7C01%7CStephen.Goss%40utdallas.edu%7C4304b8d02f9947fb8 35d08d9a91d08b2%7C8d281d1d9c4d4bf7b16e032d15de9f6c%7C0%7C0%7C637726764430593 514%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1h aWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FS84gBTOI%2Beh%2FUeGlYheeUCu7QQZS1P9ZKU987 kiQ0s%3D&reserved=0 or Android.
musliner
commented
2 years ago I think this is OBE and the revised launcher (which we did finally adopt) doesn't exhibit this
pal-tournament.zip
in game 3 we made the pogo but the game-over signal didnt get sent : a good example of how it should look is around line number 1970 ; the bad example is after 2410:
2410 PAL: b'[2021-11-15 04:32:20] [10:32:20] [BOT API THREAD/INFO]: [AGENT]CRAFT 1 minecraft:stick minecraft:stick minecraft:stick minecr aft:planks minecraft:stick minecraft:planks 0 polycraft:sack_polyisoprene_pellets 0\r\n' 2411 PAL: b'[2021-11-15 04:32:20] [10:32:20] [Client thread/INFO]: [edu.utd.minecraft.mod.polycraft.aitools.BotAPI:onClientTick:643]: CRA FT 1 minecraft:stick minecraft:stick minecraft:stick minecraft:planks minecraft:stick minecraft:planks 0 polycraft:sack_polyisoprene _pellets 0\r\n' 2412 PAL: b'[2021-11-15 04:32:20] [10:32:20] [Client thread/INFO]: Sending Command to SERVER side: edu.utd.minecraft.mod.polycraft.aitool s.commands.APICommandCraft\r\n' 2413 PAL: b'[2021-11-15 04:32:20] [10:32:20] [Server thread/INFO]: Running Command on SERVER side: edu.utd.minecraft.mod.polycraft.aitool s.commands.APICommandCraft\r\n' 2414 PAL: b'[2021-11-15 04:32:20] [10:32:20] [Server thread/INFO]: [edu.utd.minecraft.mod.polycraft.experiment.tutorial.ExperimentTutoria l:onServerTickUpdate:314]: [Server] Sending Feature update: End Condition\r\n' 2415 PAL: b'[2021-11-15 04:32:20] [10:32:20] [BOT API THREAD/INFO]: [CLIENT]{"goal":{"goalType":"ITEM","goalAchieved":true,"Distribution" :"Uninformed"},"command_result":{"command":"CRAFT","argument":"1 minecraft:stick minecraft:stick minecraft:stick minecraft:planks mi necraft:stick minecraft:planks 0 polycraft:sack_polyisoprene_pellets 0","result":"SUCCESS","message":"Crafted items: polycraft:woode n_pogo_stick","stepCost":8400.0},"step":66,"gameOver":false}\r\n' 2416 Completed game 3 2417 PAL: b'[2021-11-15 04:32:20] [10:32:20] [BOT API THREAD/INFO]: [SCORE]step:67,totalCost:92356.664,totalReward:128000.0,adjustedRewar d:35643.336\r\n' 2418 PAL: b'[2021-11-15 04:32:22] [10:32:22] [BOT API THREAD/INFO]: [edu.utd.minecraft.mod.polycraft.aitools.BotAPI$1:run:728]: SENSE_ACT OR_ACTIONS\r\n' 2419 PAL: b'[2021-11-15 04:32:22] [10:32:22] [BOT API THREAD/INFO]: [AGENT]SENSE_ACTOR_ACTIONS\r\n' 2420 PAL: b'[2021-11-15 04:32:22] [10:32:22] [Client thread/INFO]: [edu.utd.minecraft.mod.polycraft.aitools.BotAPI:onClientTick:643]: SEN SE_ACTOR_ACTIONS\r\n' 2421 PAL: b'[2021-11-15 04:32:22] [10:32:22] [Client thread/INFO]: Running Command on Client side: edu.utd.minecraft.mod.polycraft.aitool s.commands.APICommandObservation\r\n' 2422 PAL: b'[2021-11-15 04:32:22] [10:32:22] [BOT API THREAD/INFO]: [CLIENT]{"actorActions":[],"goal":{"goalType":"ITEM","goalAchieved":t rue,"Distribution":"Uninformed"},"command_result":{"command":"SENSE_ACTOR_ACTIONS","argument":"","result":"SUCCESS","message":"","st epCost":6.0},"step":67,"gameOver":false}\r\n'
this went on for another 13 minutes according to our other logs-- we sent the CHECK_COST cmd over and over and never got gameover. this pal-tournament.log doesnt show a lot of those interactions at the end, possibly due to buffering?