facebookexperimental / Robyn

Robyn is an experimental, AI/ML-powered and open sourced Marketing Mix Modeling (MMM) package from Meta Marketing Science. Our mission is to democratise modeling knowledge, inspire the industry through innovation, reduce human bias in the modeling process & build a strong open source marketing science community.
https://facebookexperimental.github.io/Robyn/
MIT License
1.13k stars 336 forks source link

Upper Boundary error in Budget Allocator #794

Closed angelinaparkina closed 5 months ago

angelinaparkina commented 1 year ago

Project Robyn

Describe issue

Hi, I am getting this error

Error in is.nloptr(ret) : at least one element in x0 > ub

when running the budget allocator with specific constraints. None of my constraints are higher than 5, which is the upper boundary, as far as I understand. Not sure if there is a separate upper boundary I am not following somehow within nloptr.

I'm providing my allocator input for now as the data is sensitive. The total budget specified is not fully allocated with the constraints specified. I tried different kinds of constraints, and the ones below do not work, but these constraints do work -

channel_constr_low = c(3.3,1.8,0.5,1.5,0.5,0.8,4.9,4.9,1,1.4),

channel_constr_up = c(3.3,1.8,0.5,1.5,0.5,0.8,4.9,4.9,1,1.4),

Also did not find anything online, so curious if someone already had this issue and knows what is going wrong.

AllocatorCollect1 <- robyn_allocator(
  InputCollect = InputCollectX,
  OutputCollect = OutputCollectX,
  select_model = select_model,
  date_range = c('2022-09-11','2022-12-31'), # Default last month as initial period
  total_budget = 2000000, # When NULL, default is total spend in date_range
  channel_constr_low = c(1.6,0.9,0.3,0.7,0.2,0.4,3.5,3.5,1,0.6),
  channel_constr_up = c(1.7,1.1,0.5,0.8,0.3,0.5,3.6,3.6,1.2,0.8),

  channel_constr_multiplier = 2,
  scenario = "max_response",
  export = FALSE
)

If it's not enough to get an answer, I will try to attach an anonimized dataset, so that it can be reproduced. Let me know.

Environment & Robyn version

Make sure you're using the latest Robyn version before you post an issue.

AdimDrewnik commented 1 year ago

I am just guessing but maybe your constraints are too narrow and within this constraints it is impossible to achieve the total budget.

angelinaparkina commented 1 year ago

Hi,

it is not possible to achieve the total budget, that is right. That is the message I get, when the budget is successfully allocated, meaning that should not stop the function to the point it cannot give me the output.

gufengzhou commented 1 year ago

Your upper and lower constraints are exactly the same, the allocator has 0 room to allocate budget. Please set lower constraints correctly, usually <1

JJohnson-DA commented 1 year ago

Does this also mean that if we want to set the constraints for a channel to be 0 on both lower and upper, that we cannot do that? Situation would be if we were spending in a channel previously that is no longer an option, but was necessary to include in building the model due to high previous spend.

mtodisco10 commented 1 year ago

I'm also running into this error. Here's an example of upper and lower constraints for a run that throws Error in is.nloptr(ret) : at least one element in x0 > ub

image

However, when I increase the channel_constr_up to the below, the allocator runs successfully. image

Can you please provide an explanation as to why this is the case?

m4x3 commented 1 year ago

Hi!

We also encountered this issue when working with the budget allocator. After digging into the functions, I think this issue is due to how the upper constraint is calculated for the unbounded scenario here: https://github.com/facebookexperimental/Robyn/blob/763b9540041e537af38d5a0ecc98c51f92f1681a/R/R/allocator.R#L306C8-L306C8 channelConstrUpSortedExt <- 1 + (channelConstrUpSorted - 1) * channel_constr_multiplier

Depending on the set channel_constraint_multiplier and the channel specific constraint inputs, the upper constraint can be lower than 0 which will raise the reported error. For a channel_constraint_multiplier of 3 the unbounded upper constraint turns negative if the upper constraint input for a channel is lower than 0.66.

@gufengzhou: I think this should be fixed. It can make sense to constrain some channels' spend to a maximum of e.g. 50% of the spend in the previous period. This should then not break the budget allocator.

The way we fixed it internally was by changing the calculation of the upper unbounded constraints to this: channelConstrUpSortedExt <- channelConstrUpSorted + (channelConstrUpSorted - channelConstrLowSorted) * channel_constr_multiplier

This always increases the upper input constraint from its original level and adds 3 times the distance between the upper and the lower input constraints. It changes a bit the logic of the channel_constr_multiplier, but that was fine for our use case.

mtodisco10 commented 1 year ago

@m4x3 can you share how you went about making adjustments to the robyn_allocator source function?

m4x3 commented 1 year ago

I suppose the easiest solution would be to try something like this: trace(name_of_function, edit = T) mentioned here and then just edit the lines you want to change.

We did it a bit differently by copying the original function code to a separate file. We then renamed the function to robyn_allocator_custom and loaded it. However, this also required to (manually) load multiple other Robyn functions and R libraries manually. It's not a great setup, but it allowed us to easily use the debug mode when executing the function. I don't know anything about package development, I'm sure there are smarter ways to work with a package's source code.

JJohnson-DA commented 1 year ago

Alternatively, if you don't want or care about the 3x results, you can simply pass 1 to the channel_constr_multiplier argument when calling the allocator.

This allows lower constraints as well as "zeroing" out channels if you don't want to spend in them for the scenario.

gufengzhou commented 1 year ago

Does this also mean that if we want to set the constraints for a channel to be 0 on both lower and upper, that we cannot do that? Situation would be if we were spending in a channel previously that is no longer an option, but was necessary to include in building the model due to high previous spend.

actually you can. setting both to 0 will exclude the channel. Just tested and it works fine image

gufengzhou commented 1 year ago

@gufengzhou: I think this should be fixed. It can make sense to constrain some channels' spend to a maximum of e.g. 50% of the spend in the previous period. This should then not break the budget allocator.

Thanks for detecting the root cause @m4x3 ! I've just pushed a fix. Let me know if it works