Use the lightweight wp_get_upload_dir() instead of wp_upload_dir().

[soderlind]

Using wp_upload_dir(), as in wp_upload_dir()['basedir'], without parameters will create a subfolder in your Uploads folder corresponding to the current month, if that folder is not already there.