Bug with higher level recipe penalty calculation for crafter levels<50 (effective level<120) results in wrong progress value

[danj2k]

I'm expecting this is not a bug, just something I am doing wrong, as I only just discovered this website. EDIT: I now believe it is a bug, see my subsequent comment below for details.

What I am seeing is that after I am inputting my level, stats etcetera, choosing a recipe and then starting the simulator, the output contains more synthesis steps than it should, and looking at the action logs it seems that this is because the progress values it is using do not match what I am seeing in-game.

The specific example I was currently trying is a level 35 Culinarian making level 40 Apple Juice. I have 164 craftsmanship, 159 control and 255 CP. I also have Careful Synthesis, Hasty Touch and Manipulation as cross-class actions.

Here's the log of a result I just got:

Seed: 1499178899141, Use Conditions: true

Monte Carlo Example
===================
 # Action                           DUR    CP      QUA      PRG    IQ  WWYW   CTL  QINC  BPRG  BQUA   WAC       Cond   S/F
 0                                   80   255        0        0     0     0   159     0     0     0     0     Normal     0
 1 Inner Quiet                       80   237        0        0     0     0   159     0     0     0     0     Normal     1
 2 Hasty Touch                       70   237        0        0     0     0   159     0     0    68     0     Normal     0
 3 Manipulation                      70   149        0        0     0     0   159     0     0     0     0     Normal     1
 4 Careful Synthesis                 70   149        0       16     0     0   159     0    16     0     0     Normal     1
 5 Steady Hand                       80   127        0       16     0     0   159     0     0     0     0     Normal     1
 6 Basic Touch                       80   109       68       16     1     0   159    68     0    68     0     Normal     1
 7 Basic Touch                       70    91      145       16     2     0   191    77     0    77     0       Good     1
 8 Hasty Touch                       60    91      273       16     3     0   223   128     0    85     0     Normal     1
 9 Hasty Touch                       50    91      367       16     4     0   254    94     0    94     0     Normal     1
10 Basic Synthesis                   40    91      367       34     4     0   286     0    18     0     0     Normal     1
11 Manipulation                      40     3      367       34     4     0   286     0     0     0     0     Normal     1
12 Careful Synthesis                 40     3      367       50     4     0   286     0    16     0     0       Good     1
13 Careful Synthesis                 40     3      367       66     4     0   286     0    16     0     0     Normal     1
14 Careful Synthesis                 40     3      367       82     4     0   286     0    16     0     0     Normal     1
15 Careful Synthesis                 30     3      367       98     4     0   286     0    16     0     0       Good     1
16 Careful Synthesis                 20     3      367      114     4     0   286     0    16     0     0     Normal     1
17 Careful Synthesis                 10     3      367      130     4     0   286     0    16     0     0     Normal     1
18 Careful Synthesis                  0     3      367      146     4     0   286     0    16     0     0     Normal     1
Progress Check: true, Durability Check: true, CP Check: true, Tricks Check: true, Reliability Check: true, Cross Class Skills: 2, Wasted Actions: 0

There are way too many synthesis actions in this, and this seems to be because it thinks the progress of Careful Synthesis on this recipe is 16. This does not match what I see in-game, which in fact is 28.

So, what have I done wrong? For example, when inputting the stats, should I have input base stats without any gear on, and then input the gear bonuses in the extra boxes on the simulator page? EDIT: I think this is a bug, see below for explanation.

[danj2k]

Oh hey, I think I figured out the problem.

This code in ffxivcraftmodel.js in the section concerning crafterLevel < 120:

        if ((levelDifference < -5)) {
            levelCorrectionFactor = 0.0501 * levelDifference;
        }
        else if ((-5 <= levelDifference) && (levelDifference < 0)) {
            levelCorrectionFactor = 0.10 * levelDifference;
        }

This is applying a greater penalty than is actually present in the game for a -5 level difference (recipe 5 levels above my crafter level). The top formula of the two calculates it correctly.

Showing working:

levelDifference = -5, craftsmanship = 164

(1 + 0.0501 * levelDifference) * (0.214959 * craftsmanship + 1.6)

=(1 + 0.0501 * -5) * (0.214959 * 164 + 1.6)

=(1 - 0.2505) * (35.253276 + 1.6)

=0.7495 * 36.853276

=27.621530362

round(27.621530362) = 28

I think there's something a bit more wrong with this code though, it doesn't make sense to me that less difficult (i.e. between 0 and -5) recipes would have a higher penalty, so fixing this issue is going to be more than just changing some operators.

EDIT: Tried to fix this in-browser using the developer console but this had unexpected results.

EDIT 2: I suggest this change? Is the penalty 10% for recipe progress between 6 and 9 levels above your crafter level, when crafter level is less than 120? That would make more sense.

        if ((levelDifference >= -5)  && (levelDifference < 0)) {
            levelCorrectionFactor = 0.0501 * levelDifference;
        }
        else if ((-5 >= levelDifference)) {
            levelCorrectionFactor = 0.10 * levelDifference;
        }

        levelCorrectedProgress = Math.round((1 + levelCorrectionFactor) * baseProgress);

[doxxx]

Is this specific to Careful Synthesis or are the other synthesis actions producing progress values that don't match the game?

[doxxx]

Given the crafting data we have received in the past, in your situation we would expect a Basic Synthesis to produce 21 progress, which would mean that Careful Synthesis should produce ~18. While this is not quite the same as the Simulator currently shows, it's close enough. So I'm very puzzled by the large difference that you're seeing.

Did you account for any food effects? You can enter these in the bonus attributes section on the right of the simulator status.

[Thortok]

You guys are aware that the formulas changed drastically in 4.0?

From the patch notes: When a recipe's level is higher than your crafter level, the difficulty of raising quality will increase, while the difficulty of raising progress will decrease.

I've also noticed smaller changes across the board but that's the biggest and the only one in the notes.

Basically, all the formulas and stuff you came up with before 4.0 are going to need tweaking.

[danj2k]

I'm now CUL 36 with craftsmanship 168 and control 170. With the same recipe I am now seeing 30 progress per Careful Synthesis, with no food effects.

Screenshot - see the bottom left where it is showing the progress values, and see the top where there are no food icons active (the ones that are active are Free Company actions, but they are just regarding EXP and should have no effect on crafting progress).

Showing working:

levelDifference = -4, craftsmanship = 168

(1 + 0.0501 * levelDifference) * (0.214959 * craftsmanship + 1.6)

=(1 + 0.0501 * -4) * (0.214959 * 168 + 1.6)

=(1 - 0.2004) * (36.113112 + 1.6)

=0.7996 * 37.713112

=30.1554043552

round(30.1554043552) = 30

EDIT: Though I guess since this is Careful Synthesis, your formula needs to be giving a result that's (1/0.9) multiplied by the above, so I guess there's a more fundamental change needed to fix this issue.

[danj2k]

Here's some data I collected this morning:

CUL 36, craftsmanship 168

Level 40 Apple Juice, progress with Basic Synthesis is 33 Level 39 Raptor Stew, progress with Basic Synthesis is 34 Level 38 Knight's Bread, progress with Basic Synthesis is 35 Level 37 Ratatouille, progress with Basic Synthesis is 36 Level 36 Cream Cheese, progress with Basic Synthesis is 37

I'll collect more data with other classes, which will have different craftsmanship due to being different levels.

[crossedxd]

I have weaver at 62 and the rest in the mid 50s, and am available to pull whatever information may be useful.

[danj2k]

More data:

GSM 25, craftsmanship 130

Level 30 Cut Stone, progress with Basic Synthesis is 25

GSM 26, craftsmanship 130

Level 26 Horn Necklace, progress with Basic Synthesis is 29

ALC 25, craftsmanship 137

Level 29 Linseed Oil, progress with Basic Synthesis is 27 Level 25 Natron, progress with Basic Synthesis is 30

I'm working on the math at the moment to try and figure out what the updated formula needs to be. Assuming that the structure of the formula is the same (because if it isn't it'll take a lot more work to figure out what it should be) then we just need to figure out the coefficients and whatnot, solving:

(1 + a levelDifference) (b * craftsmanship + c) ≅ progress

for a, b and c. The difficulty here is the progress numbers we see are rounded from what the formula outputs.

[doxxx]

We used an Excel spreadsheet with initially our own data when we were still playing FFXIV and later with data submitted by helpful players, and ran Excel's solver on it to determine the coefficients. The game numbers being rounded is a significant hindrance to improving the accuracy.

It requires a lot of data and time to nail down the formula accurately enough to be useful. Neither of us have the free time or the game access to do this work and verify its accuracy. Updating recipe lists is mostly automated, which is why I have been able to (mostly) keep up with that.

If somebody can figure out the formula, I'll gladly update the code to use it. @Ermad is doing some great work in PR #139 to incorporate the various new actions and effects.

[crossedxd]

Any chance you still have the Excel spreadsheet somewhere, or could give me an idea/template of what I need to pull together to at least kick off the solver process? I'd gladly do the grunt work involved to get the tool back on its feet, considering how much I've leveraged it since I started playing again.

[danj2k]

@doxxx I'd also be interested in how that Excel spreadsheet works, I've not used Excel's solver before - I was actually planning on just doing the math by hand - so that would speed up the process.

[danj2k]

@crossedxd to refine the formula to fix this specific issue more data is needed:

Crafting class and level Recipe name and level Craftsmanship stat (don't forget this changes depending what equipment you have on) Amount of progress per Basic Synthesis (ensure you aren't affected by any Food bonuses, or any other bonuses to progress that might be granted by other abilities)

I managed to brute force a possible range of values that can be plugged in to the formula but the more 4.0 data that's available, the, more this can be narrowed down until I can reach a single set of values.

Including recipes that are the same level as you is useful, as this gives a levelDifference of zero which effectively eliminates the first term of the formula.

Including a variety of recipes that have different level differences and different craftsmanship stats is also useful.

[lokyst]

@danj2k I've posted a link to the spreadsheet on the Wiki.

[danj2k]

@lokyst Thanks! Though looking at it I have to say it is considerably over my head.

[lokyst]

Gist of it is to use the inputs (craftsmanship, level etc.) to calculate the progress based on your formula, compare that to the actual result, and then use the Excel solver to minimize the sum of the squared errors by allowing it to change the coefficients of your formula (I've typically put these near the top right of the tab -- you may need to scroll right a lot). Ideally you want the errors to be under 1 point, but because the numbers are truncated this is not always possible.

Squeenix had different formulas for recipes below your level, even level and higher level. In addition, the formulas varied between ARR recipes and Heavensward -- hence all the tabs.

Adding ingenuity into the mix should only be considered once you nail down the base formula(s).

[crossedxd]

I ran some tests this evening to pull a spread of progression numbers. All synth actions were performed by a Lv.62 weaver, using the Ruby Cotton Yarn recipe (also Lv.62).

Craftmanship	C. Synth	Basic Synth	C. Synth 2	Standard Synth
356	64	72	86	108
364	66	73	88	110
365	66	73	88	110
373	68	75	90	113
387	70	78	94	117
395	72	80	96	120
396	72	80	96	120
403	73	81	98	122
404	73	81	98	122
411	75	83	100	125
412	75	83	100	125
420	76	85	102	127
434	79	88	105	132
442	80	89	107	134
443	80	89	107	134
444	81	90	108	135
451	82	91	109	137
452	82	91	109	137
453	82	92	110	138
461	84	93	112	140
475	86	96	115	144
483	88	98	117	147
484	88	98	118	147
491	90	100	120	150
492	90	100	120	150
499	91	101	122	152
500	91	101	122	152
508	93	103	124	155
522	95	106	127	159
530	97	108	129	162
531	97	108	130	162
539	99	110	132	165

[Ermad]

Thanks, I was able to get the base progress formula for Stormblood recipes from that.

Biggest thing now is to gather data for recipes of a different level then your crafter level.

[crossedxd]

I can run more tests if needed, though the cheaper the better. Is there anything specific you would like me to test?

[Ermad]

Just need data for as many level differences as possible (both above and below). Don't need as many data points per level difference though, just 2 different craftsmanship values with those 4 actions should suffice.

[crossedxd]

I'll work on pulling that now, then. The ridiculous number of data points above was just because I was attempting my own regression work with it, but I don't have any real stats experience, so I'd rather leave that to those who do. :)

[crossedxd]

Alright, here's what I've managed to pull together so far. Note that this was again accomplished using a Lv.62 weaver. I don't have access to anything above Lv.65 so far, so unfortunately that's as high as I can go.

Recipe Name	Recipe Level	iLvl (ffxivcrafting)	Craftsmanship	C. Synth	Basic Synth	C. Synth 2	Standard Synth
Ramie Thread	54	133	539	155	172	(complete)	(complete)
Hallowed Ramie Cloth	56	139	539	155	172	207	(complete)
Crawler Silk	57	142	539	155	172	206	(complete)
Chimerical Felt	58	145	539	155	172	206	(complete)
Chimerical Felt Chausses of Healing	59	145	539	154	172	206	258
Bloodhempen Yarn	61	255	539	134	148	178	223
Ruby Cotton Yarn	62	265	539	99	110	132	165
Ruby Cotton Cap	63	270	539	86	95	114	143
Kudzu Thread	64	273	539	79	87	105	131
Kudzu Cap of Crafting	65	276	539	74	82	98	123
Ramie Thread	54	133	309	88	98	117	147
Hallowed Ramie Cloth	56	139	309	88	98	117	147
Crawler Silk	57	142	309	88	97	117	146
Chimerical Felt	58	145	309	88	97	117	146
Chimerical Felt Chausses of Healing	59	145	309	87	97	117	146
Bloodhempen Yarn	61	255	309	76	84	101	126
Ruby Cotton Yarn	62	265	309	56	62	75	93
Ruby Cotton Cap	63	270	309	48	54	65	81
Kudzu Thread	64	273	309	44	49	59	74
Kudzu Cap of Crafting	65	276	309	42	46	56	70

[Ermad]

Thanks. Would it be possible to get values for recipes even higher level difference, or is that the max you are able to craft?

[crossedxd]

65 is my current max with weaver at 62. I can grind a bit tomorrow and see if I can unlock the other recipes on weaver, but if something in the 50-60 range is acceptable I can just look into getting the data from one of my other classes.

[Ermad]

I pushed an update to my branch (in #139), that has this taken into account. It should be decently accurate now, besides recipes above your level.

Now need more different level difference data points for recipes above your level. Have -5, -8, and -11 from your data so far. All my crafters are level 70, so I am only able to get data for -10 (1 star recipes), which I will do tomorrow.

Also need to check the Ingenuity 1 & II values. The recipes ilvl scaled in same exact fashion from 61-70 as they did from 51-60 in Heavensward. So currently it just scaled the Ingenuity values by the same amount, need to do testing if this is actually accurate.

[Ermad]

Also need data for quality increase when using touches on recipes above your level. My crafters are all 70 so I can only get it for -10 level difference.

[crossedxd]

Here are my current class levels (some may be close to leveling up): ALC - 53 ARM - 52 BSM - 52 CRP - 51 CUL - 52 GSM - 54 LTW - 52 WVR - 62

If you have any ideas of what specifically you'd like me to pull data-wise, let me know. I can also look into it a bit tomorrow after work.

[Ermad]

Once you get to Weaver to 63 it will open up more. Doing recipes for the 3 levels above you will give -3, -6, -9 then.

[crossedxd]

I'll do that tomorrow, then. Material costs may be an issue, but I'll cross that bridge when I come to it.

[Ermad]

Thanks a ton! If you were on Cactuar I could help out, but doubt that you are.

The quality increase formula from Heavensward actually stills seems accurate (within a couple points) for recipes at or below your level. The penalty for recipes above your level has changed though, if you are able to record the quality values of a few touches at the same time.

[crossedxd]

I'll grab that info as well. Right now I just run a pretty simple info-gathering macro against each recipe (Reclaim, SH II, the 4x synths, and spam synth till I fail). I can just do the same for the touches when I'm info-gathering tomorrow (basic, standard, advanced).

[danj2k]

Sounds like @Ermad and @crossedxd have crafter level 50+/60+ well in hand, so I will focus on crafter levels <50 since that's where I'm at:

31 Carpenter 29 Blacksmith n/a Armorer 26 Goldsmith 37 Leatherworker 32 Weaver 26 Alchemist 36 Culinarian

(I will pick up Armorer, it's just I haven't really needed it for anything until now).

[Ermad]

Was doing some testing on it myself, it seems like the higher recipe penalty is just -2.5% on progress for each recipe level capped at 25%, and quality is -5% for each recipe level capped at -50%.

Ingenuity (I & II) seem to follow the same pattern as in Heavensward, so I am happy with their current values unless someone can show some conflicting data.

[crossedxd]

So what does that mean from a data collection standpoint? Do you still need me to pull the synth/touch data we were discussing earlier, or would it make more sense for me to just test the solver out and see how it compares to what I see in-game?

[Ermad]

Having data for the other level difference values would be good to confirm it, but if its a lot of trouble it should be fine without it.

Would also be nice to get some Heavensward data (crafter level 51-60) to see if that changed (both equal level and below recipe level), while you level those up. Or at least can you see if the values in simulator or still accurate for those.

[crossedxd]

What I'll do when I get home is just put together a list of stuff to test and just do it all at once. Right now I'm planning on running synth/touch tests for Lv.64-70 recipes with my Lv.63 weaver, and then run a -10/+10 spread for synth/touch on my Lv.54 goldsmith (or go as high as I can; is 10 too much of a delta?). Let me know if that should suffice, or if there's anything else that would help out. (This data will look like the 2nd table I posted in this thread.)

[Ermad]

That looks good to me, thanks again.

[danj2k]

I picked up Armorer and gathered some data this evening, but I'm not having any luck figuring out the constants for recipes below crafter level (where crafter level is < 50).

crafterClass	crafterLevel	recipeName	recipeLevel	levelDifference	craftsmanship	progress
ARM	1	Bronze Ingot	1	0	39	10
ARM	2	Bronze Ingot	1	1	39	10
ARM	2	Bronze Rings	2	0	39	10
ARM	3	Bronze Ingot	1	2	39	11
ARM	3	Bronze Plate	3	0	39	10
ARM	4	Bronze Hoplon	5	-1	39	9
ARM	6	Bronze Hoplon	5	1	39	10
ARM	6	Bronze Rivets	2	4	65	18
ARM	6	Bronze Alembic	3	3	65	18
ARM	7	Bronze Ingot	1	6	65	19
ARM	7	Bronze Rings	2	5	65	19
ARM	7	Bronze Chain Coif	8	-1	65	15
ARM	8	Bronze Ingot	1	7	65	20

[Ermad]

Hard to do much data that has that little craftsmanship, since so much information is lost by the rounding. Would be better to get as much data as you can from your level 30s/40s crafter instead.

[crossedxd]

So at Lv.63, it appears as though I only have recipes up to 68 available to me. Do we really need anything greater than a +5/-5 split for testing purposes?

[Ermad]

That should be good, I think its capped at a penalty for 10 recipe levels above and +5 is 15 recipe levels above.

[crossedxd]

Alright, so here are my results for Lv.63-68 recipes using a Lv.63 weaver. I've included the four synth actions from before, and also added basic/standard/advanced touch actions and the Control stat. I'm going to forego generating the goldsmithing data I mentioned earlier unless you absolutely need it, since I want to actually play the game a little bit tonight. XD

Recipe Name	Recipe Level	iLvl (ffxivcrafting)	Craftsmanship	Control	Basic Touch	Standard Touch	Adv. Touch	C. Synth	Basic Synth	C. Synth 2	Standard Synth
Ruby Cotton Cap	63	270	539	528	210	262	315	98	109	131	164
Kudzu Thread	64	273	539	528	178	223	267	91	101	121	151
Kudzu Cap of Crafting	65	276	539	528	146	183	220	84	93	112	140
Worsted Yarn	66	279	539	528	115	144	173	76	84	101	126
Serge Turban of Crafting	67	282	539	528	104	131	157	74	82	98	123
Twinthread	68	285	539	528	104	130	157	74	82	98	123
Ruby Cotton Cap	63	270	390	356	150	187	225	71	78	94	118
Kudzu Thread	64	273	390	356	127	159	191	65	72	87	108
Kudzu Cap of Crafting	65	276	390	356	104	131	157	60	67	80	100
Worsted Yarn	66	279	390	356	82	103	123	54	60	72	91
Serge Turban of Crafting	67	282	390	356	74	93	112	53	59	70	88
Twinthread	68	285	390	356	74	93	112	53	59	70	88

[Ermad]

Thanks for the data. That confirms for me that progress gets a 2.5% penalty per recipe level up to max 25%, and quality gets a 5% penalty per recipe level up to max of 50%.

I think this change to penalties also applies crafters of all level from the new data that I have seen of lower levels.

[crossedxd]

Based on that comment referring to the patch notes, I'd say that's a safe assumption to make.

[danj2k]

So as you suggested I collected some data with my 30+ crafters, but I'm still having trouble figuring out the new coefficients for recipes below the crafter level:

crafterClass	crafterLevel	recipeName	recipeLevel	levelDifference	craftsmanship	progress
CRP	31	Steel Spear	30	1	159	37
CRP	31	Walnut Cane	29	2	159	39
CRP	31	Elm Macuahuitl	28	3	159	40
CRP	31	Silver Battle Fork	27	4	159	42
CRP	31	Viper-crested Round Shield	26	5	159	44
CUL	36	Herring Ball	35	1	176	41
CUL	36	Acorn Cookie	34	2	176	43
CUL	36	Tuna Miq'abob	33	3	176	45
CUL	36	Forest Miq'abob	32	4	176	47
CUL	36	Dark Vinegar	31	5	176	48
LTW	37	Boarskin Ringbands	36	1	168	39
LTW	37	Boarskin Survival Boots	35	2	168	41
LTW	37	Boarskin Choker	34	3	168	43
LTW	37	Boarskin Himantes	33	4	168	44
LTW	37	Toadskin Belt	32	5	168	46

I'll keep trying, but I think the formula's structure may have changed for recipes below the crafter level where the crafter level is <50. Unfortunately if that's the case, I don't have the capability to figure out a whole new formula.

With assistance from stackoverflow I now have some python code which I can use for least squares regression which gives me a formula of the form:

progress = Math.round(a * levelDifference + b * craftsmanship + c)

(I realise that least squares regression is also what that big Excel spreadsheet is doing, but while I tried to understand how to do it in Excel I could not grasp it. With the Python code I don't need to understand much, the hardest part was actually figuring out how to put the data in there)

I successfully found using the stackoverflow code a set of values for a, b and c that fit 1 <= levelDifference <= 5, I'm assuming that as with the original code, different values will be needed going forward to larger values of levelDifference, so I will collect more data.~~

It seems like you guys were already on top of this as I see there are new changes.

[Ermad]

That seems accurate to whats already in the simulator, to within a point. The only thing that has seemed to have changed is the penalty for recipes above your level, which we know and looks consistent across all craft levels.

[danj2k]

@Ermad I'm not sure if I tested it incorrectly but when I tried, I could not get the existing code in the simulator to output values that matched the data within the required +/- 0.5 margin of error. With the new formula that I have, I can achieve that.

EDIT: as I'm adding more data I am finding it difficult to get the error margin down to that level. So yeah I don't know.

As to how I've been testing it, I forked the project on github (though the fork has now been deleted as I could not see a way to get it to accept the new changes that @Ermad and @doxxx have put in) then cloned it to my home Linux box. The github.io idea seems better though as that can be safely linked to other people.

[crossedxd]

@Ermad is the version of the simulator you have up and running up-to-date with the most recent values? I'm going to be doing some crafting today and figured I could cross-reference my results with what the simulator gives me.

[doxxx]

@danj2k I have not updated either the main or the beta site with Ermad's changes yet.

[Ermad]

You can test my branch out at https://ermad.github.io/ffxiv-craft-opt-web/app/

[doxxx]

Awesome! I didn't know you could do that!