search

azfeng8 / GLIB-Baking-Fails-and-LLMs

0 stars 0 forks source link

PDDLGym v0.0.6 causes LNDR to perform worse #3

Open azfeng8 opened 4 months ago

azfeng8 commented 4 months ago

The new version of pddlgym (0.0.6) causes a performance bug: LNDR-GLIB performance drastically decreases in Baking and Travel, but not in Minecraft.

Found that warmstart performance on commit f37f9bc was drastically worse on Baking and Travel than 2 commits earlier (ada429f) performance.

To find the bug, I ran experiments on a branch of f37f9bc after changing APIs to v0.0.5 standard and bumping pddlgym version to 0.0.5. The performance for Baking increased to as seen before (ada429f).

azfeng8 commented 4 months ago

Searching for the cause of the bug:

Commented out the contains addition to LiteralSpace in the most recent pddlgym version. Still has performance decrease for LNDR, so that is not the source of the problem.

Ran seeds 80-90 of LNDR on the modified branch:

https://github.com/azfeng8/pddlgym/tree/remove-passive-warnings

image