High Resolution Extraction of maps and/or vmaps

[adellaci]

🚀 Feature

High Resolution Extraction of maps and/or vmaps I modified the ExtractResources.sh to add the option to add hi rez maps & vmaps extraction, and to restore MoveMapGen8Core.sh to manage executing of MoveMapGen with the use of all available processors up to 8 cores.

I've tested it. I would like confirmation, and someone to submit it.

Code ExtractResources.sh

#!/bin/sh

# This file is part of the CMaNGOS Project. See AUTHORS file for Copyright information
#
# This file is free software; as a special exception the author gives
# unlimited permission to copy and/or distribute it, with or without
# modifications, as long as this notice is preserved.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY, to the extent permitted by law; without even the
# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

## Expected param 1 to be 'a' for all, else ask some questions

## Normal log file (if not overwritten by second param
LOG_FILE="MaNGOSExtractor.log"
## Detailed log file
DETAIL_LOG_FILE="MaNGOSExtractor_detailed.log"

## Change this to a value fitting for your sys!
NUM_CPU="2"

## ! Use below only for finetuning or if you know what you are doing !

USE_AD="0"
USE_VMAPS="0"
USE_MMAPS="0"
USE_MMAPS_OFFMESH="0"
USE_MMAPS_DELAY=""
AD_REZ=""
VMAP_REZ=""

if [ "$1" = "a" ]
then
  ## extract all
  USE_AD="1"
  USE_VMAPS="1"
  USE_MMAPS="1"
  USE_MMAPS_DELAY="no"
else
  ## do some questioning!
  echo
  echo "Welcome to helper script to extract required dataz for MaNGOS!"
  echo "Should all dataz (dbc, maps, vmaps and mmaps be extracted? (y/n)"
  read line
  if [ "$line" = "y" ]
  then
    ## extract all
    USE_AD="1"
    USE_VMAPS="1"
    USE_MMAPS="1"
  else
    echo
    echo "Should dbc and maps be extracted? (y/n)"
    read line
    if [ "$line" = "y" ]; then USE_AD="1"; fi

    echo
    echo "Should vmaps be extracted? (y/n)"
    read line
    if [ "$line" = "y" ]; then USE_VMAPS="1"; fi

    echo
    echo "Should mmaps be extracted? (y/n)"
    echo "WARNING! This will take several hours! (you can later tell to start delayed)"
    read line
    if [ "$line" = "y" ]
    then
      USE_MMAPS="1";
    else
      echo "Only reextract offmesh tiles for mmaps?"
      read line
      if [ "$line" = "y" ]
      then
        USE_MMAPS_OFFMESH="1";
      fi
    fi
  fi
fi

## Special case: Only reextract offmesh tiles
if [ "$USE_MMAPS_OFFMESH" = "1" ]
then
  echo "Only extracting offmesh meshes"
  MoveMapGen.sh offmesh $LOG_FILE $DETAIL_LOG_FILE
  exit 0
fi

## MMap Extraction specific
if [ "$USE_MMAPS" = "1" ]
then
  ## Obtain number of processes
  echo "How many CPUs should be used for extracting mmaps? (1, 2, 4, 8)"
  read line
  echo
  if [ $line -eq 1 ] || [ $line -eq 2 ] || [ $line -eq 4 ] || [ $line -eq 8 ]; then
    NUM_CPU=$line
  else
    echo "Only numbers 1,2,4 and 8 are supported!"
    exit 1
  fi
  ## Extract MMaps delayed?
  if [ "$USE_MMAPS_DELAY" != "no" ]; then
    echo "MMap extraction can be started delayed"
    echo "If you do _not_ want MMap Extraction to start delayed, just press return"
    echo "Else enter number followed by s for seconds, m for minutes, h for hours"
    echo "Example: \"3h\" - will start mmap extraction in 3 hours"
    read -p"MMap Extraction Delay (leave blank for direct extraction): " USE_MMAPS_DELAY
    echo
  else
    USE_MMAPS_DELAY=""
  fi
fi

## Check if user want to do high resolution extraction of maps
if [ "$USE_AD" = "1" ]; then
  echo
  echo "Would you like to do hi-rez extraction of maps? (y/n)"
  read line
  if [ "$line" = "y" ]; then
    AD_REZ="-f 0"
  else
    AD_REZ=""
  fi
fi

## Check if user want to do high resolution extraction of vmaps
if [ "$USE_VMAPS" = "1" ]; then
  echo
  echo "Would you like to do hi-rez extraction of vmaps? (y/n)"
  read line
  if [ "$line" = "y" ]; then 
    VMAP_REZ="-l"
  else
    VMAP_REZ=""
  fi
fi
## Give some status
echo "Current Settings: Extract DBCs/maps: $USE_AD, Extract vmaps: $USE_VMAPS, Extract mmaps: $USE_MMAPS on $NUM_CPU processes"
echo
if [ "$USE_AD" = "1" ]; then echo "./ad $AD_REZ"; fi
if [ "$USE_VMAPS" = "1" ]; then echo "./vmaps_extractor $VMAP_REZ"; fi
if [ "$USE_MMAPS_DELAY" != "" ]; then
  echo "MMap Extraction will be started delayed by $USE_MMAPS_DELAY"
fi
echo
if [ "$1" != "a" ]
then
  echo "If you don't like this settings, interrupt with CTRL+C"
  read line
fi

echo "`date`: Start extracting dataz for MaNGOS" | tee $LOG_FILE

## Handle log messages
if [ "$USE_AD" = "1" ];
then
  echo "DBC and map files will be extracted" | tee -a $LOG_FILE
else
  echo "DBC and map files won't be extracted!" | tee -a $LOG_FILE
fi
if [ "$USE_VMAPS" = "1" ]
then
  echo "Vmaps will be extracted" | tee -a $LOG_FILE
else
  echo "Vmaps won't be extracted!" | tee -a $LOG_FILE
fi
if [ "$USE_MMAPS" = "1" ]
then
  echo "Mmaps will be extracted with $NUM_CPU processes" | tee -a $LOG_FILE
else
  echo "Mmaps files won't be extracted!" | tee -a $LOG_FILE
fi
echo | tee -a $LOG_FILE

echo "`date`: Start extracting dataz for MaNGOS, DBCs/maps $USE_AD, vmaps $USE_VMAPS, mmaps $USE_MMAPS on $NUM_CPU processes" | tee $DETAIL_LOG_FILE
echo | tee -a $DETAIL_LOG_FILE

## Extract dbcs and maps
if [ "$USE_AD" = "1" ]
then
 echo "`date`: Start extraction of DBCs and map files..." | tee -a $LOG_FILE
 ./ad $AD_REZ | tee -a $DETAIL_LOG_FILE
 echo "`date`: Extracting of DBCs and map files finished" | tee -a $LOG_FILE
 echo | tee -a $LOG_FILE
 echo | tee -a $DETAIL_LOG_FILE
fi

## Extract vmaps
if [ "$USE_VMAPS" = "1" ]
then
  echo "`date`: Start extraction of vmaps..." | tee -a $LOG_FILE
  ./vmap_extractor $VMAP_REZ | tee -a $DETAIL_LOG_FILE
  echo "`date`: Extracting of vmaps finished" | tee -a $LOG_FILE
  mkdir vmaps
  echo "`date`: Start assembling of vmaps..." | tee -a $LOG_FILE
  ./vmap_assembler Buildings vmaps | tee -a $DETAIL_LOG_FILE
  echo "`date`: Assembling of vmaps finished" | tee -a $LOG_FILE

  echo | tee -a $LOG_FILE
  echo | tee -a $DETAIL_LOG_FILE
fi

## Extract mmaps
if [ "$USE_MMAPS" = "1" ]
then
  if [ "$USE_MMAPS_DELAY" != "" ]; then
    echo "Extracting of MMaps is set to be started delayed by $USE_MMAPS_DELAY"
    echo "Current time: $(date)"
    sleep $USE_MMAPS_DELAY
  fi
  sh MoveMapGen.sh $NUM_CPU $LOG_FILE $DETAIL_LOG_FILE
fi>

Code MoveMapGen8Core.sh

#!/bin/sh
echo date
# This file is part of the CMaNGOS Project. See AUTHORS file for Copyright information
#
# This file is free software; as a special exception the author gives
# unlimited permission to copy and/or distribute it, with or without
# modifications, as long as this notice is preserved.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY, to the extent permitted by law; without even the
# implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

## Syntax of this helper
## First param must be number of to be used CPUs (only 1, 2, 4, 8 supported) or "offmesh" to recreate the special tiles from the OFFMESH_FILE
## Second param can be an additional filename for storing log
## Third param can be an addition filename for storing detailed log

## Additional Parameters to be forwarded to MoveMapGen, see mmaps/readme for instructions
PARAMS="--silent"

## Already a few map extracted, and don't care anymore
EXCLUDE_MAPS=""
#EXCLUDE_MAPS="0 1 530 571" # example to exclude the continents

## Offmesh file
OFFMESH_FILE="offmesh.txt"

## Normal log file (if not overwritten by second param
LOG_FILE="MoveMapGen.log"
## Detailed log file
DETAIL_LOG_FILE="MoveMapGen_detailed.log"

## ! Use below only for finetuning or if you know what you are doing !

## All maps
LIST_A="571"
LIST_B="1"
LIST_C="530"
LIST_D="0"
LIST_E="169 531 409 545 604 542 44 109 389 565 450 617 616 592 582 489"
LIST_F="603 631 574 560 269 602 309 585 566 469 25 615 369 230 649 576 547 555 48 35 129 558 557 249 449 588 586 608"
LIST_G="607 533 509 543 30 600 532 650 553 43 554 34"
LIST_H="619 628 723 609 564 37 534 595 209 568 529 573 658 47 349 229 546 189 552 540 598 556 601 544 593 42 451 587"
LIST_I="591 548 33"
LIST_J="594 572 562 329"
LIST_K="596 580 429 36 13"
LIST_L="289 724 559 599 668 90 550 70 632 624 618 590 584 589 578 575"

badParam()
{
  echo "ERROR! Bad arguments!"
  echo "You can (re)extract mmaps with this helper script,"
  echo "or recreate only the tiles from the offmash file"
  echo
  echo "Call with number of processes (1,2,4,8) to create mmaps"
  echo "Call with 'offmesh' to reextract the tiles from offmash file"
  echo "Example: ./MoveMapGen.sh 8"
  echo
  echo "For further fine-tuning edit this helper script"
  echo
}

if [ "$#" = "3" ]
then
  LOG_FILE=$2
  DETAIL_LOG_FILE=$3
elif [ "$#" = "2" ]
then
  LOG_FILE=$2
fi

# Offmesh file provided?
OFFMESH=""
if [ "$OFFMESH_FILE" != "" ]
then
  if [ ! -f "$OFFMESH_FILE" ]
  then
    echo "ERROR! Offmesh file $OFFMESH_FILE could not be found."
    echo "Provide valid file or none. You need to edit the script"
    exit 1
  else
    OFFMESH="--offMeshInput $OFFMESH_FILE"
  fi
fi

# Function to process a list
createMMaps()
{
  for i in $@
  do
    for j in $EXCLUDE_MAPS
    do
      if [ "$i" = "$j" ]
      then
        continue 2
      fi
    done
    ./MoveMapGen $PARAMS $OFFMESH $i | tee -a $DETAIL_LOG_FILE
    echo "`date`: (Re)created map $i" | tee -a $LOG_FILE
  done
}

createHeader()
{
  echo "`date`: Start creating MoveMaps" | tee -a $LOG_FILE
  echo "Used params: $PARAMS $OFFMESH" | tee -a $LOG_FILE
  echo "Detailed log can be found in $DETAIL_LOG_FILE" | tee -a $LOG_FILE
  echo "Start creating MoveMaps" | tee -a $DETAIL_LOG_FILE
  echo
  echo "Be PATIENT - This will take a long time and might also have gaps between visible changes on the console."
  echo "WAIT until you are informed that 'creating MoveMaps' is 'finished'!"
}

# Create mmaps directory if not exist
if [ ! -d mmaps ]
then
  mkdir mmaps
fi

# Param control
case "$1" in
  "1" )
    createHeader $1
    createMMaps $LIST_A $LIST_B $LIST_C $LIST_D $LIST_E $LIST_F $LIST_G $LIST_H $LIST_I $LIST_J $LIST_K $LIST_L &
    ;;
  "2" )
    createHeader $1
    createMMaps $LIST_A $LIST_C $LIST_E $LIST_G $LIST_J $LIST_L &
    createMMaps $LIST_B $LIST_D $LIST_F $LIST_H $LIST_I $LIST_K &
    ;;
  "4" )
    createHeader $1
    createMMaps $LIST_A $LIST_E &
    createMMaps $LIST_B $LIST_F $LIST_I &
    createMMaps $LIST_C $LIST_G $LIST_J $LIST_L &
    createMMaps $LIST_D $LIST_H $LIST_K &
    ;;
  "8" )
    createHeader $1
    createMMaps $LIST_A &
    createMMaps $LIST_B &
    createMMaps $LIST_C &
    createMMaps $LIST_D &
    createMMaps $LIST_E $LIST_I $LIST_J $LIST_K &
    createMMaps $LIST_F &
    createMMaps $LIST_G $LIST_L &
    createMMaps $LIST_H &
    ;;
  "offmesh" )
    echo "`date`: Recreate offmeshs from file $OFFMESH_FILE" | tee -a $LOG_FILE
    echo "Recreate offmeshs from file $OFFMESH_FILE" | tee -a $DETAIL_LOG_FILE
    while read map tile line
    do
      ./MoveMapGen $PARAMS $OFFMESH $map --tile $tile | tee -a $DETAIL_LOG_FILE
      echo "`date`: Recreated $map $tile from $OFFMESH_FILE" | tee -a $LOG_FILE
    done < $OFFMESH_FILE &
    ;;
  * )
    badParam
    exit 1
    ;;
esac

wait

echo  | tee -a $LOG_FILE
echo  | tee -a $DETAIL_LOG_FILE
echo "`date`: Finished creating MoveMaps" | tee -a $LOG_FILE
echo "`date`: Finished creating MoveMaps" >> $DETAIL_LOG_FILE
echo date

[Phatcat]

Instead of opening an issue you should rather open a pull-request against one of the cores

[adellaci]

@Phatcat I do not know how to do a pull request, at least with out making a mess of things.

[adellaci]

@Phatcat i'll give it a shot again, but i would still like feed back on this.

[ghost]

It's quite hard to provide feedback on diffs without diffs ^_~

Pull requests aren't hard: you need to fork target repo to your own account, do the changes, push to your fork and then press the magic green button to request a pull.

That being said, no special git CLI magic involved, just your usual git workflow, the PR and fork -related stuff is handled through graphical interfaces of GitHub.

[ghost]

Check it out: https://help.github.com/articles/about-pull-requests/

[Muehe]

@adellaci Just FYI, the existing script already supports using up to 8 cores.

[adellaci]

@Muehe I did try it without using MoveMapGen8Core.sh but it error with a wrong parameter fault.

[adellaci]

I tried a pull-request but i believe i have failed again.

[Muehe]

How did you start it? And what was your error message exactly?

[AnonXS]

This branch is 5 commits ahead, 2802 commits behind cmangos:master. https://github.com/adellaci/mangos-wotlk/commit/2043679d601237420ac82cb35c7d6771c9e7c32b

i dont feel well about your fork being 2800+ commits behind origin/master.

you should create a development branch on your fork, push your changes there and keep your fork udpated with origin/master.

[adellaci]

@AnonXS Is there a way for me to delete my old one, and the weekend i'll jump on Discord to if someone is available to assist me in the correct way to do it?

[adellaci]

@Muehe When get home it will rerun it and post the error

[Muehe]

There are several ways to fix this, and we can help you with it on Discord. That being said, the files have been moved in the meantime, so the easiest solution would probably be to re-fork and manually apply the changes again.

[adellaci]

@AnonXS adellaci/mangos-wotlk@2043679 This one is very old one, on the original work i had done to reorganize the map processing distribution to increase the speed of map processing, and add up to 8 processors. I initially was going to add more processor, but my research / testing resulted that anything more then 8 processor had no impact on reducing the time to process. That one has been commit and is the current script being used (I believe) and should have been close.

It was during that contribution that I messed things up trying to do it with pull request. I can not remember who it was that created it on my behalf, because of the mess i made. I'm not even sure if i ever thanked the individual that did that for me. <--- what an ungrateful "B" i am..LOL.

All that aside, I do want to learn the feature/function of 'git'. I started last night looking up tutorials, but all i have found, assumed that you understand the process some what. I know zero. I can clone & pull, but that is all i know.

[adellaci]

@Muehe When i first started writing the script change this was the first error ./ExtractResources.sh ........ sh: 0: Can't open MoveMapGen.sh When i looked. MoveMapGen did not have a '.sh' extension. so i changed the following line sh MoveMapGen.sh $NUM_CPU $LOG_FILE $DETAIL_LOG_FILE to sh MoveMapGen $NUM_CPU $LOG_FILE $DETAIL_LOG_FILE then reran with this error MoveMapGen: 9: MoveMapGen: Syntax error: ")" unexpected It was at that point i changed it to; sh ./MoveMapGen8Core.sh $NUM_CPU $LOG_FILE $DETAIL_LOG_FILE This worked as intended, so that is were i ended up in the change of the extractor script.

[ulfgebhardt]

@adellaci Try this article: https://reflectoring.io/github-fork-and-pull/

[Muehe]

@adellaci I have taken the liberty to just recreate your script changes on an up-to-date wotlk fork: https://github.com/Muehe/mangos-wotlk/tree/extract

I also changed the script shebang from #!/bin/sh to #!/bin/bash, like it is in the classic/tbc repo (has nothing to do with your changes).

Once I have checked that the script still works, I will cherry pick your commit (used the author info from your old commits btw., hope that's OK) into the other cores and open PRs.

As for learning to work with git, I can recommend this website: https://learngitbranching.js.org/

It has an interactive shell on the website and guides you through some common actions when working with git. Hands down the best introduction I found. If you are looking for additional info, some more links may be found in our wiki.

[adellaci]

@ulfgebhardt Thank you very much for the info. @Muehe Thank you also for the info, and the commit, I do not mind at all.

[anyone31]

Great.

Hint:

"#!/bin/bash" -> "#!/bin/sh" "AD_REZ" -> "AD_RES" "VMAP_REZ" -> "VMAP_RES" "hi-rez extraction" -> "high resolution extraction" or "high-res extraction"

[Muehe]

@anyone31 I included your suggestions into my PRs, except the change from bash to sh. Why did you suggest that? I intentionally changed it on WotLK, because Classic and TBC already used bash.

Example of current output:

$ ./ExtractResources.sh         

Welcome to helper script to extract required dataz for MaNGOS!
Should all dataz (dbc, maps, vmaps and mmaps be extracted? (y/n)
y
How many CPUs should be used for extracting mmaps? (1, 2, 4, 8)
4

MMap extraction can be started delayed
If you do _not_ want MMap Extraction to start delayed, just press return
Else enter number followed by s for seconds, m for minutes, h for hours
Example: "3h" - will start mmap extraction in 3 hours

MMap Extraction Delay (leave blank for direct extraction):
1h

Would you like the extraction of maps to be high-resolution? (y/n)
y

Would you like the extraction of vmaps to be high-resolution? (y/n)
y

Current Settings:
Extract DBCs/maps: 1, Extract vmaps: 1, Extract mmaps: 1, Processes for mmaps: 4
maps extraction will be high-resolution
vmaps extraction will be high-resolution
MMap Extraction will be started delayed by 1h

Press (Enter) to continue, or interrupt with (CTRL+C)

[RussianE39]

bash not always should be available at /bin/bash. Some platforms have it at /usr/local/bin/bash

[adellaci]

@Muehe first thank you for handling the github stuff, and @RussianE39 and you for your inputs.

On the topic of bash vs sh, reminded me on the work i added to "InstallFullDB.sh". We change that from shebang sh to bash in order to get this section of the script to display correctly. I do not remember the exact reason why it impacted it. That said, i do not see any part of this script that would impact how it displays or functions.

if [ "$FORCE_WAIT" != "NO" ]
then
  echo "ATTENTION: Your database $DATABASE will be reset to WoTLK-DB!"
  echo "Please bring your repositories up-to-date!"
  echo "Press CTRL+C to exit"
  # show a mini progress bar
  for i in {1..10}
  do
   echo -ne .
   sleep 1
  done
  echo .
fi```

[adellaci]

i posted results of all three trunk in the respective trunk. In the results of all three extractions are similar time wise. It was the same as when I did testing on my research changes to MoveMapGen.sh for 8 cores. Still again map 1 is the longest to process. In looking back at what my results were i did discover who had done the pull and commit for me.

So thanks to @cyberium & @Muehe for both of your help on both of those.

[adellaci]

As soon as i get a chance i'm going to enroll at github lab to train on how to do this myself.

[Muehe]

@anyone31 @RussianE39 Well, the script was actually using bash syntax until this PR was merged recently. The shebang being different on Wotlk was probably why it was made in the first place.

I have made the change to sh on the PRs and in addition checked the script with this web tool. They define different warnings, I included fixes for SC2006 (change `date` to $(date), cause POSIX) and ignored SC2086 and SC2162, since the strings used do not include backslashes/spaces/newlines/etc.

@adellaci For understanding Github PRs, you have to understand git branches/remotes first, once you understand that PRs are trivial. I recommend https://learngitbranching.js.org/ again, it helped me a lot in that regard.

[Muehe]

Merged. Thanks @adellaci and @anyone31 :+1: