Statblocks Bestiary Formatting Update

[sigrunixia]

@djbrock or @Craftidore

The script used to pull the data.

I am hoping one of you may be able to take point here. We're about to release a major update to IT Builder, and there is some file formatting issues with the bestiary that prevents IT from reading it. Specifically, there are about 1000 files that need to be re-pulled and wrapped to contain their special characters.

For example in Monster Babba Yaga.

layout: Basic Pathfinder 1e Layout
source: "Pathfinder No. 72: The Witch Queen's Revenge"
Monster_CR: 30
name: Baba Yaga
Monster_XP: 9830400
race: Female
class: venerable advanced human witch 20, archmage 10 (Pathfinder RPG Advanced Player’s Guide 65, Pathfinder RPG Bestiary 294, Pathfinder RPG Mythic Adventures 14)
alignment: NE
size: Medium
type: humanoid
subtype: (human, mythic)
INI: +19
perception: +27
senses: arcane sight, darkvision, see invisibility
AC: 51, touch 31, flat-footed 46 (armor +13, deflection +5, dex +5, insight +11, natural +7)
HP: 782
HP_extra: regeneration 20
HD: 20d6+710
saves: Fort +30, Ref +27, Will +35
immune: ability damage and drain, charm and compulsion effects, death effects, disease, energy drain, petrification, poison, all forms of madness
DR: 10/epic and good
defensive_abilities: hard to kill, mythic saving throws, unstoppable
SR: 35 (arcane only)
speed: 30 ft.
melee: Baba Yaga’s besom +17/+12 (1d6+7)
special_attacks: archmage arcana (wild arcana), channel power, coupled arcana, energy conversion, forceful will, hexes (agony [20 rounds], cackle [swift action 2/day], cauldron, child scent, cook people, death curse, evil eye [-4, 21 rounds], scar, life giver, slumber [20 rounds], witch’s hut, vision), mythic hexes, mythic power (23/ day, surge +1d12), mythic presence (DC 28), reverse scrying, sanctum, throw spell
pf1e_stats: [17, 20, 26, 46, 24, 32]
BAB: 10
CMB: 13
CMD: 36
feats: Accursed Critical, Accursed Hex, Brew Potion, Extra Hex, Extra Path Ability, Greater Spell Penetration, Improved Familiar, Improved Initiative, Mythic Paragon, Mythic Spell Lore, Quicken Spell, Spell Penetration, Split Hex, Split Major Hex, Toughness
skills: Acrobatics +15, Appraise +23, Bluff +26, Craft (alchemy) +22, Craft (woodworking) +22, Diplomacy +21, Escape Artist +25, Fly +23, Heal +25, Intimidate +36, Knowledge (arcana) +41, Knowledge (dungeoneering) +38, Knowledge (engineering) +38, Knowledge (geography) +38, Knowledge (history) +41, Knowledge (local) +38, Knowledge (nature) +41, Knowledge (nobility) +38, Knowledge (planes) +41, Knowledge (religion) +38, Perception +27, Ride +13, Sense Motive +27, Spellcraft +41, Stealth +25, Survival +11, Use Magic Device +34
languages: Abyssal, Aklo, Aquan, Auran, Celestial, Common, Draconic, Giant, Ignan, Infernal, Iobarian, Polish, Russian, Sarmatian, Sylvan, Terran, Triaxian, tongues
special_qualities: amazing initiative (+10), component freedom, contingency, crafting mastery, enduring armor, exceptional stats, forceful presence, hut familiar, immortal, legendary hero, long-lived, mythic spell power, permanent spells, Queen of Witches, recuperation, true archmage
gear:
  - name: gear
    desc: amulet of natural armor +5, apple of eternal sleep, Baba Yaga’s besom (see page 62), belt of mighty constitution +6, bracelet of second chances, cackling hag’s blouse, corset of dire witchcraft, ebony fly figurine of wondrous power, gem of seeing, Grandmother’s basket (as a portable hole), Grandmother’s shawl (as a cloak of resistance +5), grim lantern, hat of disguise, haunted shoes, headband of alluring charisma +6, hexing doll, orb of foul Abaddon, pearl of power (5th level), ring of elemental command (fire), ring of freedom of movement (slotless), ring of protection +5; in addition to the gear listed here, Baba Yaga has access to a wide variety of other magic items that she can use at the GM’s discretion.
special_abilities:
  - name: Contingency
    desc: Whenever Baba Yaga is attacked, she is affected by mislead.
  - name: Exceptional Stats (Ex)
    desc: Baba Yaga was born a perfect specimen of humanity. As a result, her ability scores were generated using 25 points, rather than using the standard 15 point buy used to create most NPCs. Additionally, Baba Yaga has much more gear than an NPC of her level would normally have. These modifications increase her total CR by 2.
  - name: Forceful Presence (Ex)
    desc: Baba Yaga adds her Charisma modifier as an insight bonus to her Armor Class and on all of her saving throws. In addition, Baba Yaga uses her Charisma score as well as her Constitution score when calculating hit points. This ability increases her total CR by 1.
  - name: Hut Familiar (Ex)
    desc: Baba Yaga built her Dancing Hut using a combination of the witch’s hut hex and the sanctum mythic path ability, and over time expanded the artifact’s powers to its present state. The Dancing Hut of Baba Yaga serves as Baba Yaga’s familiar, and replaces the normal witch’s familiar ability. Baba Yaga’s life is forever intertwined with her Dancing Hut and parts of her essence reside within it. As long as the hut exists, this special connection grants her 300 bonus hit points, DR 10/epic and good, and regeneration 20. This ability increases her total CR by 1.
  - name: Immortal (Su)
    desc: Baba Yaga has hidden her death to make herself even more difficult to kill. This functions as the base mythic ability (Mythic Adventures 13), but Baba Yaga always returns to life 24 hours later, regardless of how she is killed. In order to permanently kill Baba Yaga, her death must be found and released back into her body, at which point she can be killed with a coup de grace or critical hit made with an artifact.
  - name: Long-Lived (Ex)
    desc: Mystical arcane powers, mythic abilities, and complex arcane rituals have extended Baba Yaga’s life. She gains the +3 bonus to Intelligence, Wisdom, and Charisma scores for having lived beyond venerable age, but she doesn’t gain the penalties to Strength, Dexterity, and Constitution resulting from this advanced age. Baba Yaga’s supernaturally empowered life grants her complete immunity to ability damage and drain, charm and compulsion effects, death effects, disease, energy drain, petrification, poison, and all forms of madness (including confusion effects and feeblemind). This ability increases her total CR by 1.
  - name: Permanent Spells
    desc: Through the use of permanency, Baba Yaga is constantly under the effect of the following spells: arcane sight, darkvision, read magic, see invisibility, and tongues. Baba Yaga has also devised a way to inscribe symbol spells upon her person. Baba Yaga has permanent symbols of death (DC 36) magically inscribed on the center of her chest, between her shoulder blades, upon the crown of her head, and on the soles of her feet. Only a few great beings have knowledge of the symbols’ passwords (each symbol has its own unique phrase), and even fewer significant personalities have been attuned to the symbols.
  - name: Queen of Witches (Ex)
    desc: As Queen of Witches, Baba Yaga knows all witch spells, as well as all sorcerer/wizard spells. Baba Yaga also has knowledge of many other spells that she has researched. Many of these are arcane versions of divine spells. In addition, Baba Yaga can create artifacts, and has done so to great extent, the greatest of which is the Dancing Hut of Baba Yaga.
spells_prepared:
  - name:
    desc: (CL 20)
  - name: 9th
    desc: dominate monster (DC37), foresight, gate, time stop, wail of the banshee (DC37), weird (DC37), wish
  - name: 8th
    desc: binding (DC36), quickened dimension door, maze, quickened phantasmal killer (DC36), polymorph any object (DC36), stormbolts (DC36), symbol of death (DC36)
  - name: 7th
    desc: control weather, finger of death (DC35), greater teleport, heal, jolting portent (DC35), terraform, temporary resurrection
  - name: 6th
    desc: quickened blindness/deafness (DC34), chain lightning (DC34), disintegrate (DC34), eyebite (DC34), greater dispel magic, slay living (DC34), true seeing
  - name: 5th
    desc: baleful polymorph (DC33), break enchantment, dominate person (DC33), feeblemind (DC33), greater forbid action (DC33), major curse (DC33), smug narcissism, steal power (DC33)
  - name: 4th
    desc: bestow curse (DC32), charm monster (DC32), confusion (DC32), dimensional anchor, ice storm, greater invisibility, stoneskin, terrible remorse (DC32)
  - name: 3rd
    desc: cackling skull (DC31), dispel magic, fireball (DC31), fly, haste, lightning bolt (DC31), protection from energy, suggestion (DC31)
  - name: 2nd
    desc: anticipate peril (DC30), glitterdust (DC30), mad hallucination (DC30), mirror image, miserable pity (DC30), pox pustules (DC30), touch of idiocy, web (DC30)
  - name: 1st
    desc: beguiling gift (DC30), bungle (DC30), charm person (DC29), disguise self (DC29), feather fall, fumbletongue (DC30), 2xmagic missile
  - name: 0 (at-will)
    desc: daze (DC28), mending, prestidigitation, read magic
sources:
  - name: Pathfinder No. 72: The Witch Queen's Revenge
    desc: 74

needs to become

```statblock
layout: Basic Pathfinder 1e Layout
source: "Pathfinder No. 72: The Witch Queen's Revenge"
Monster_CR: 30
name: Baba Yaga
Monster_XP: 9830400
race: Female
class: "venerable advanced human witch 20, archmage 10 (Pathfinder RPG Advanced Player’s Guide 65, Pathfinder RPG Bestiary 294, Pathfinder RPG Mythic Adventures 14)"
alignment: NE
size: Medium
type: humanoid
subtype: "(human, mythic)"
INI: +19
perception: +27
senses: "arcane sight, darkvision, see invisibility"
AC: 51, touch 31, flat-footed 46 (armor +13, deflection +5, dex +5, insight +11, natural +7)"
HP: 782
HP_extra: regeneration 20
HD: 20d6+710
saves: "Fort +30, Ref +27, Will +35"
immune: "ability damage and drain, charm and compulsion effects, death effects, disease, energy drain, petrification, poison, all forms of madness"
DR: "10/epic and good"
defensive_abilities: "hard to kill, mythic saving throws, unstoppable"
SR: "35 (arcane only)"
speed: "30 ft."
melee: "Baba Yaga’s besom +17/+12 (1d6+7)"
special_attacks: "archmage arcana (wild arcana), channel power, coupled arcana, energy conversion, forceful will, hexes (agony [20 rounds], cackle [swift action 2/day], cauldron, child scent, cook people, death curse, evil eye [-4, 21 rounds], scar, life giver, slumber [20 rounds], witch’s hut, vision), mythic hexes, mythic power (23/ day, surge +1d12), mythic presence (DC 28), reverse scrying, sanctum, throw spell"
pf1e_stats: [17, 20, 26, 46, 24, 32]
BAB: 10
CMB: 13
CMD: 36
feats: "Accursed Critical, Accursed Hex, Brew Potion, Extra Hex, Extra Path Ability, Greater Spell Penetration, Improved Familiar, Improved Initiative, Mythic Paragon, Mythic Spell Lore, Quicken Spell, Spell Penetration, Split Hex, Split Major Hex, Toughness"
skills: "Acrobatics +15, Appraise +23, Bluff +26, Craft (alchemy) +22, Craft (woodworking) +22, Diplomacy +21, Escape Artist +25, Fly +23, Heal +25, Intimidate +36, Knowledge (arcana) +41, Knowledge (dungeoneering) +38, Knowledge (engineering) +38, Knowledge (geography) +38, Knowledge (history) +41, Knowledge (local) +38, Knowledge (nature) +41, Knowledge (nobility) +38, Knowledge (planes) +41, Knowledge (religion) +38, Perception +27, Ride +13, Sense Motive +27, Spellcraft +41, Stealth +25, Survival +11, Use Magic Device +34"
languages: "Abyssal, Aklo, Aquan, Auran, Celestial, Common, Draconic, Giant, Ignan, Infernal, Iobarian, Polish, Russian, Sarmatian, Sylvan, Terran, Triaxian, tongues"
special_qualities: "amazing initiative (+10), component freedom, contingency, crafting mastery, enduring armor, exceptional stats, forceful presence, hut familiar, immortal, legendary hero, long-lived, mythic spell power, permanent spells, Queen of Witches, recuperation, true archmage"
gear:
  - name: gear
    desc: "amulet of natural armor +5, apple of eternal sleep, Baba Yaga’s besom (see page 62), belt of mighty constitution +6, bracelet of second chances, cackling hag’s blouse, corset of dire witchcraft, ebony fly figurine of wondrous power, gem of seeing, Grandmother’s basket (as a portable hole), Grandmother’s shawl (as a cloak of resistance +5), grim lantern, hat of disguise, haunted shoes, headband of alluring charisma +6, hexing doll, orb of foul Abaddon, pearl of power (5th level), ring of elemental command (fire), ring of freedom of movement (slotless), ring of protection +5; in addition to the gear listed here, Baba Yaga has access to a wide variety of other magic items that she can use at the GM’s discretion."
special_abilities:
  - name: Contingency
    desc: "Whenever Baba Yaga is attacked, she is affected by mislead."
  - name: "Exceptional Stats (Ex)"
    desc: "Baba Yaga was born a perfect specimen of humanity. As a result, her ability scores were generated using 25 points, rather than using the standard 15 point buy used to create most NPCs. Additionally, Baba Yaga has much more gear than an NPC of her level would normally have. These modifications increase her total CR by 2."
  - name: "Forceful Presence (Ex)"
    desc: "Baba Yaga adds her Charisma modifier as an insight bonus to her Armor Class and on all of her saving throws. In addition, Baba Yaga uses her Charisma score as well as her Constitution score when calculating hit points. This ability increases her total CR by 1."
  - name: "Hut Familiar (Ex)"
    desc: "Baba Yaga built her Dancing Hut using a combination of the witch’s hut hex and the sanctum mythic path ability, and over time expanded the artifact’s powers to its present state. The Dancing Hut of Baba Yaga serves as Baba Yaga’s familiar, and replaces the normal witch’s familiar ability. Baba Yaga’s life is forever intertwined with her Dancing Hut and parts of her essence reside within it. As long as the hut exists, this special connection grants her 300 bonus hit points, DR 10/epic and good, and regeneration 20. This ability increases her total CR by 1."
  - name: "Immortal (Su)"
    desc: "Baba Yaga has hidden her death to make herself even more difficult to kill. This functions as the base mythic ability (Mythic Adventures 13), but Baba Yaga always returns to life 24 hours later, regardless of how she is killed. In order to permanently kill Baba Yaga, her death must be found and released back into her body, at which point she can be killed with a coup de grace or critical hit made with an artifact."
  - name: "Long-Lived (Ex)"
    desc: "Mystical arcane powers, mythic abilities, and complex arcane rituals have extended Baba Yaga’s life. She gains the +3 bonus to Intelligence, Wisdom, and Charisma scores for having lived beyond venerable age, but she doesn’t gain the penalties to Strength, Dexterity, and Constitution resulting from this advanced age. Baba Yaga’s supernaturally empowered life grants her complete immunity to ability damage and drain, charm and compulsion effects, death effects, disease, energy drain, petrification, poison, and all forms of madness (including confusion effects and feeblemind). This ability increases her total CR by 1."
  - name: Permanent Spells
    desc: "Through the use of permanency, Baba Yaga is constantly under the effect of the following spells: arcane sight, darkvision, read magic, see invisibility, and tongues. Baba Yaga has also devised a way to inscribe symbol spells upon her person. Baba Yaga has permanent symbols of death (DC 36) magically inscribed on the center of her chest, between her shoulder blades, upon the crown of her head, and on the soles of her feet. Only a few great beings have knowledge of the symbols’ passwords (each symbol has its own unique phrase), and even fewer significant personalities have been attuned to the symbols."
  - name: "Queen of Witches (Ex)"
    desc: "As Queen of Witches, Baba Yaga knows all witch spells, as well as all sorcerer/wizard spells. Baba Yaga also has knowledge of many other spells that she has researched. Many of these are arcane versions of divine spells. In addition, Baba Yaga can create artifacts, and has done so to great extent, the greatest of which is the Dancing Hut of Baba Yaga."
spells_prepared:
  - name:
    desc: "(CL 20)"
  - name: 9th
    desc: "dominate monster (DC37), foresight, gate, time stop, wail of the banshee (DC37), weird (DC37), wish"
  - name: 8th
    desc: "binding (DC36), quickened dimension door, maze, quickened phantasmal killer (DC36), polymorph any object (DC36), stormbolts (DC36), symbol of death (DC36)"
  - name: 7th
    desc: "control weather, finger of death (DC35), greater teleport, heal, jolting portent (DC35), terraform, temporary resurrection"
  - name: 6th
    desc: "quickened blindness/deafness (DC34), chain lightning (DC34), disintegrate (DC34), eyebite (DC34), greater dispel magic, slay living (DC34), true seeing"
  - name: 5th
    desc: "baleful polymorph (DC33), break enchantment, dominate person (DC33), feeblemind (DC33), greater forbid action (DC33), major curse (DC33), smug narcissism, steal power (DC33)"
  - name: 4th
    desc: "bestow curse (DC32), charm monster (DC32), confusion (DC32), dimensional anchor, ice storm, greater invisibility, stoneskin, terrible remorse (DC32)"
  - name: 3rd
    desc: "cackling skull (DC31), dispel magic, fireball (DC31), fly, haste, lightning bolt (DC31), protection from energy, suggestion (DC31)"
  - name: 2nd
    desc: "anticipate peril (DC30), glitterdust (DC30), mad hallucination (DC30), mirror image, miserable pity (DC30), pox pustules (DC30), touch of idiocy, web (DC30)"
  - name: 1st
    desc: "beguiling gift (DC30), bungle (DC30), charm person (DC29), disguise self (DC29), feather fall, fumbletongue (DC30), 2xmagic missile"
  - name: "0 (at-will)"
    desc: "daze (DC28), mending, prestidigitation, read magic"
sources:
  - name: "Pathfinder No. 72: The Witch Queen's Revenge"
    desc: 74

[djbrock]

I've actually been tinkering with this, as I'd noticed that some don't render correctly. That was specifically for 'special_abilities' though. I'll try and replicate what I did for all of the other items changed above and recreate the markdown for all of the monsters.

What I'm doing is:

s=str(monster_in['special_abilities'][sa]).encode('unicode_escape').decode('ASCII')
print('    desc:', '"'+s+'"', file=f)

If anyone knows a better way - shout please. What's the pythonic way to write a string that may contain newlines and other things so it gets correctly escaped?

[Craftidore]

I'll have to check how yaml likes it's newlines escaped, but I took the current and intended outputs and put them in a diff checker.

It appears that there are some places where the values have special yaml characters, and quotes have to be used to escape them. It seems like the easiest way to solve this is to wrap every string in quotes (which doesn't hurt anything in yaml) and only bothering to check for quotes to escape.

[Craftidore]

(I haven't used ttrpg statblocks before, so if there are quirks to how it works with yaml I wouldn't know about them)

[sigrunixia]

The rule of thumb is if it has a special character, wrap it in a quote (or bracket style if using brackets, but quotes are more consistent).

While regular YAML is fine with the YAML as listed, Statblocks is doing an absurd amount of regex to search for dice rollers, spells, etc. The special characters unescaped and somethimes not wrapped in quotes breaks its ability to parse out what it should be rendering. Really, we expect statblocks to do too much, but it is what it is.

[Craftidore]

Wouldn't the best solution still be to wrap all values in quotes?

[sigrunixia]

Some integers are fine, but it seems more and more a safe solution if its not a table array, is to just wrap everything. 👍🏻