Automatising manual processes

[Sazaju]

In order to make a dedicated program to automatise some manual stuff, I need you to tell me the different tasks to achieve. For each of them, I would need one or several representative examples with:

• what you start from
• what you expect to have at the end
• why you need such a change

By having the two first points, I can implement automatic tests to guarantee that it works as intended. The last point helps me to understand the requirements of the task so that I can generalize the example.

We can go one task after the other, or you may give me all the tasks at once, as you want.

[Sazaju]

I have started to implement something. So far, given an existing Japanese file and an existing English file, it creates 2 new files : the new English file aligned with the Japanese + the obsolete English entries. So from:

ScriptTextEnglish.rvtext
ScriptTextJapanese.rvtext

I obtain:

ScriptTextEnglish.rvtext
ScriptTextEnglish.rvtext.new
ScriptTextEnglish.rvtext.unused
ScriptTextJapanese.rvtext

The original files are not touched. The "new" file is strictly aligned to the Japanese version (same entries in the same order) such that when an English entry is available, it is used, otherwise this is the Japanese entry which is used. The "unused" file contains all the English entries of the original file which cannot be mapped to an entry in the Japanese file.

If you make a diff between the Japanese file and the "new" file, you would see that only the content of the entries is modified (so they are actually in the same order):

And in order for you to have an idea of the steps done and the time it takes, here is the short log:

2015-09-03 04:57:55.315 INFO: Loading Japanese file MGQ-EX/ScriptTextJapanese.rvtext... [fr.sazaju.mgqupdater.MGQUpdater.<init>()]
2015-09-03 04:57:55.790 INFO: Reading MGQ-EX/ScriptTextJapanese.rvtext... [fr.vergne.translation.impl.PatternFileMap.startIterator()]
2015-09-03 04:57:57.555 INFO: Japanese loaded: 74309 entries. [fr.sazaju.mgqupdater.MGQUpdater.<init>()]
2015-09-03 04:57:57.556 INFO: Loading English file MGQ-EX/ScriptTextEnglish.rvtext... [fr.sazaju.mgqupdater.MGQUpdater.<init>()]
2015-09-03 04:57:57.557 INFO: Reading MGQ-EX/ScriptTextEnglish.rvtext... [fr.vergne.translation.impl.PatternFileMap.startIterator()]
2015-09-03 04:57:59.024 INFO: English loaded: 74309 entries. [fr.sazaju.mgqupdater.MGQUpdater.<init>()]
2015-09-03 04:57:59.025 INFO: Writing new English file MGQ-EX/ScriptTextEnglish.rvtext.new... [fr.sazaju.mgqupdater.MGQUpdater.<init>()]
2015-09-03 04:58:01.416 INFO: New English file written: 274 entries remaining [fr.sazaju.mgqupdater.MGQUpdater.<init>()]
2015-09-03 04:58:01.417 INFO: Writing unused English file MGQ-EX/ScriptTextEnglish.rvtext.unused... [fr.sazaju.mgqupdater.MGQUpdater.<init>()]
2015-09-03 04:58:01.426 INFO: Unused English file written: 274 entries [fr.sazaju.mgqupdater.MGQUpdater.<init>()]

Japanese loading: 2.24s English loading: 1.47s New English creation: 2.40s Unused English creation : 0.01s

I see that some entries have additional codes in English (e.g. ルカ becomes \n[1]), so I assume there is additional treatments to do. But for this kind of treatment, I would expect to add some seconds to retrieve/replace/add codes before each writing. Even if you have several treatments like that, it may take like 30s or a minute (if you have a lot). The only kind of treatment I see which could need a lot of time would be treatment that you need to make in several passes, although I would not expect it to take more than few minutes (but it surely depends on the task).

[MGQ-EX]

Seconds I don't think much on, Minutes start getting noticeable, Hours is when you go out to eat/sleep/enjoy life then come back.

ScriptText is a custom build for entries not natively handled by LFS, and LFS currently has no native sorting functions so all current sorting is by algorithms I implemented and am still working on though bugs that effect people currently using this take priority especially game breaking ones. It will be nice though to test out the full capabilities of your editor.

\n[X] are entries that pull actor names from the Database file and are being implemented to make management easier. Typos, bad translations, etc should be easier to fix as one central entry instead of thousands of individual entries, however some overlap which is somewhat intentional and will be dealt with later (some actor names share enemy names but this also adds functionality within the game), please don't overwrite them is the only treatment however if you want to overlay the real data then: From Database \n[x] - Writes actor x's name. actors/X/name \en[x] - Writes out enemy's name. enemies/X/name \nc[x] - Writes out class x's name. classes/X/name \ni[x] - Writes out item x's name. items/X/name \nw[x] - Writes out weapon x's name. weapons/X/name \na[x] - Writes out armour x's name. armors/X/name \ns[x] - Writes out skill x's name. skills/X/name \nt[x] - Writes out state x's name. states/X/name \ne[x] - Writes out element x's name. types/elements/X

The Icons are pulled from the game, don't worry about them \ii[x] - Writes out item x's name including icon. items/X/name \iw[x] - Writes out weapon x's name including icon. weapons/X/name \ia[x] - Writes out armour x's name including icon. armors/X/name \is[x] - Writes out skill x's name including icon. skills/X/name \it[x] - Writes out state x's name or icon. states/X/name \ie[x] - Writes out element x's name or icon. types/elements/X

These require nothing additional as they are formatting within the game \n - Creates a name box with x string. Left side. \nc - Creates a name box with x string. Centered. \nr - Creates a name box with x string. Right side. \v[x] - Writes variable x's value. \g - Writes gold currency name. \i[x] - Draws icon x at position of the text.

[Sazaju]

What I would like is really to align on your process: I have understood that you use LFS to extract the rvtexts from the game, excepted ScriptText, and re-apply them to the game. My point is not to re-implement LFS, but to see all the additional stuff that you do, in order to improve the automation of this part. Do you have your own scripts or are you using a cutomized LFS?

[Sazaju]

Can you make a picture of your process? If there is a lot of steps it may be easier to get the point for me with a picture. You can just use Paint or any basic drawing tool on your computer, or you can also use Dia to make a clean diagram (I use this one).

[Sazaju]

Or you can draw by hand and take a photo too.

[MGQ-EX]

How is this flow chart

[Sazaju]

I don't really get what happens with 3, in particular I don't get what is the link between RPG Maker and LFS (what happens between 2 and 4). Don't you use LFS directly on the files of the game? Do you change something to the game itself (with RPG Maker) before to apply LFS? And what do you mean by applying LFS "in game"? While the game is running? Another confusing thing is that it mixes with translation tasks: 3 uses a translation script and 6 translates internal scripts. Are they proper JP-EN translations or other kinds of translations? Do you translate stuff which is not in the database/dialogue/script files? And what are the errors you speak about in 10? Are they translation errors or bugs occurring because of some coding mistakes?

Maybe you don't need to answer (all) this questions. So far, here is the process I understood, where I separate the tasks from the inputs/outputs (JP = Japanese, EN = English):

Am I correct so far? If not, what should be fixed?

[MGQ-EX]

1) LFS is an internal script, only functions in game and is not independent. It runs within the game, alters it, and then uses the altered files as an overlay; Yep it actually consumes the game from the inside out to make this work 2) Yes as LFS script needs to be added to the game in order to function 3) After it is added to the game 4) The game has to run an extraction process call to LFS script during run time 5) Internal script items are not handled by LFS, it mainly extracts Database & Dialogue entries; A custom process was made to extract Script items though it still is a work in progress and doesn't get everything 6) Yes mostly menu commands that are programmed in a way that aren't easily extracted, variables & constants if found are added to the Script extraction. Custom Translation Script handles some of the odd entries, mostly is a set of arrays that gain entries the other two don't handle. Place Names being an example as the Game actually stores these as symbols and converts them to strings before displaying them, this also overrides the script that does that and instead will pull the data from ScriptText then display it properly 7) As proper as currently possible 8) Repeat of question 6 altered slightly 9) Errors are errors with the entire project... Be it translation, coding either introduced or native, and user related. So all of the above, and are addressed on a case by case basis. 10) So far so good, All the functional components seem to be there and if you have any more questions feel free I appreciate your help 11) I'm sure a lot could be fixed and I'm probably a mad man... All jokes aside fixes are dealt with in question 9 as anything that needs fixed is probably an error of some sort to improve either logic or design

[Sazaju]

OK. So it seems that a significant amount of tasks is deeply interacting with the game itself. I don't plan to work on that kind of tasks, so here is the main point for now: if my diagram is correct, then the English RVText files (database/dialogues/scripts) are completely separated from the RPG Maker and LFS tasks. Only the Japanese RVText files are impacted. Then you have additional tasks (compare + apply) dedicated to the English RVText files. If it is wrong, then don't read further and please tell me how the English files are impacted by RPG Maker or LFS.

If it is correct, then the main task for me now is to integrate all the tasks you do to adapt the English RVText files based on the Japanese ones. For now, my code is made to take the new Japanese file + the old English file in order to produce the corresponding new English file. Thus, two things may miss:

• consider the old JP RVText
• consider the files specificities (database/dialogues/scripts)

First, let's consider the generic stuff: does the old JP RVText provide useful information? For instance, it may be that JP entries see there IDs changed, so we need to rely on their content (or other stuff) to recognize them. In such a case, comparing the old and new may be indeed required. If it does not happen (e.g. they may be removed, new entries may be added, but even if they have the same content they come from a different context), then everything can be inferred from the new version alone. A different way to ask is the following: the comparison task building on the new JP + old JP, if we assume that the old JP and the old EN are fully aligned (EN = JP with translated content), could you do the same thing that you do now by using the old EN instead of the old JP (so new JP + old EN)? If the answer is yes, then what I am doing now is, a priori, enough. If the answer is (or could be) no, then I need you to detail the "Compare" task. When I say details, I don't mean algorithmic details, only requirement details (what you start from, what you get once it is done, and why you need it that way).

Second, regarding the specificities, I need to know two kinds of them:

• the specific properties (when I consider a given type of file, what should I do with it)
• the specific interactions (when I consider a given type of file, what should I do with files of other types)

For both of them, the details needed are for both "Compare" and "Apply" tasks.

[MGQ-EX]

1) Diagram looks correct 2) Surprised you actually considered the update process so going far beyond that is out of scope so no problems there 3) Currently the New JP is compared with the Old JP for a few items

• What currently matches
• What changed
• What is new
• What was removed
• What shifted/duplicated

From this: Current Matches don't change so the English one isn't effected Changed Entries, retain a record of changed entries as sometime very little changes but it still needs reviewed New Entries get added Removed Entries get deleted The last one just tries to salvage anything from the old version before discarding it Entries get sorted then saved as new files As those are task RPG Maker wasn't designed for this kind of work it is a relatively slow process (Hours) though it is currently used as the function task are there and operational.
4) It's not a good idea to rely on the New JP to ever match the Old EN, the entries can shift drastically if even a little thing is changed, actual dev team changes a map name then all information dealing with that map gets shifted even if very little inside it changed or a copy of a map is made and given a new ID which even if it is the same map it gets flagged as new content. That was fun with 1.21 5) Only file types in this are rvtext files, perhaps a txt file for a saved report, and a few internal update algorithms which handle update task ".rvupd", ".rvadd", and ".rvdel" but aren't more then rvtext files with less data currently used to see how things changed, translate changes if the size isn't to large, and integrate changes.