facebookexperimental / Robyn

Robyn is an experimental, AI/ML-powered and open sourced Marketing Mix Modeling (MMM) package from Meta Marketing Science. Our mission is to democratise modeling knowledge, inspire the industry through innovation, reduce human bias in the modeling process & build a strong open source marketing science community.
https://facebookexperimental.github.io/Robyn/
MIT License
1.13k stars 334 forks source link

Unable to achieve Target ROAS with Python Robyn API Allocator Function - target_efficiency #931

Open apoorvabc opened 7 months ago

apoorvabc commented 7 months ago

I am working with Python Robyn API. Upon executing the allocator function for the "target_efficiency" scenario, it consistently generates results based on the default target ROAS value, which corresponds to 80% of the initial ROAS.

In the below scenario, despite setting the target ROAS value to 2 in the allocator function, the output does not reflect this specified target. In the below image you can see that the desired target ROAS value was not achieved as intended, despite attempting to set it to 2.

allocatorArgs = {
    "select_model": select_model,
    "date_range": ('2023-03-03', '2023-09-09'), 
    "target_value ": 2,
    "scenario": "target_efficiency"
}
payload = {
    'InputCollect': json.dumps(InputCollect),
    'OutputCollect': json.dumps(OutputCollect),
    "jsonAllocatorArgs": json.dumps(allocatorArgs),
    'dpi': 100, 
    'width': 15,
    'height': 15
}
robyn_api('robyn_allocator', payload=payload)

image

yu-ya-tanaka commented 7 months ago

Hi @apoorvabc, Can you use an R version library to check if the result will be the same? I think it's not a Robyn API issue, but it may be impossible to create a scenario to achieve a ROAS of 200% with given data and model.

apoorvabc commented 7 months ago

I have tried running the allocator on windows as well as Linux machines with different R versions and still the same issue. I lowered the target_value to be less than 1 and I still get the same response as above.

Also, to add to that I get different outputs for the max_reponse on Linux and windows for the same payload parameters.

Windows : R version 4.3.1 , Linux : R version 4.3.2 Robyn Version: ‘3.10.3’

apoorvabc commented 5 months ago

Hello, I still have the same issue. Could you please help me understand what might be wrong?

yu-ya-tanaka commented 5 months ago

Sorry for being late. I was busy organising events. Let me clarify the situation. Do you have the issue only when running Python Robyn API or do you have the issue both Python Robyn API and R?

apoorvabc commented 5 months ago

I have so far worked with just Python Robyn API and the issue persists.

yu-ya-tanaka commented 5 months ago

Can you do the same thing with the same data in R to identify if this is an API issue or a Robyn issue in R? Also, as there have been several updates since 3.10.3, could you update the Robyn version to latest and try again?

gufengzhou commented 5 months ago

First of all, target ROAS is not always achievable, for example your dataset might never achieve ROAS of 100. For your case of target 2, please try widening the arguments channel_constr_low and channel_constr_up and see if it gets closer to 2.

apoorvabc commented 5 months ago

What is the default value it takes when I do not enter any channel constraints in the case of "target_efficiency"?

I was under the impression that the budget constraints need to be added only in case of "max_response" and not for "target_efficiency". For example, in the above scenario I have entered only the target ROAS but with no constraints for the channels.

gufengzhou commented 4 months ago

For target_efficiency, default are 0.1 for channel_constr_low and Inf for channel_constr_up. In your case, it looks like the allocator won't find higher ROAS by scaling up spend (it's already 9x spend). Maybe try setting lower constraint to close to 0?