search

sxjscience / automl_multimodal_benchmark

Repository for Multimodal AutoML Benchmark
60 stars 7 forks source link

'Price' column included in airbnb dataset #5

Open jpgard opened 1 year ago

jpgard commented 1 year ago

Hi, thanks for the great benchmark!

I wanted to ask about something I noticed -- this seems to be a bug, but not quite positive. It looks like the 'price' column in the airbnb dataset, which is used to generate the 'price_label' column (which is a binned prediction target) is included in the input features. This means that 'price' can be used to predict 'price_label' -- which should allow for perfect accuracy for a moderately-flexible classifier from the price column alone.

https://github.com/sxjscience/automl_multimodal_benchmark/blob/c3478b1f595eeb593f367de42196a31dbf223e1d/multimodal_text_benchmark/scripts/data_processing/process_airbnb.py

Is this intended? And if so, maybe I am also unclear why the baseline models in the paper aren't getting 100% on this dataset?

sxjscience commented 1 year ago

Sorry for the late reply as I just see this issue.

The "price" column has been marked as "ignored" in https://github.com/sxjscience/automl_multimodal_benchmark/blob/77edc53ef7e8a91ea4cfdaa336b353af66e3db00/multimodal_text_benchmark/src/auto_mm_bench/datasets.py#L480-L485 .