read_csv s3 file parameter memory_map default is True?

[autocyz]

Code Sample

# code 1
def get_pid():
    info = pd.read_csv('s3://vision.algo.data/cyz/tmp/test/11.csv', header=None, names=['pid'])
    print(f'pid num : {len(info)}')

# code 2
def get_pid():
    info = pd.read_csv('s3://vision.algo.data/cyz/tmp/test/11.csv', header=None, names=['pid'], memory_map=False)
    print(f'pid num : {len(info)}')

while True:
    get_pid()
    time.sleep(4)

Problem description

when I use code1, if s3 file changed, info not change. code2 can find the change. The difference between these two code is parameter memory_map, actually, default memory_map is False, so I was confused.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit : None python : 3.7.5.final.0 python-bits : 64 OS : Linux OS-release : 4.4.0-1098-aws machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 0.25.3 numpy : 1.17.4 pytz : 2019.3 dateutil : 2.8.0 pip : 19.3.1 setuptools : 42.0.2 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : 2.8.4 (dt dec pq3 ext lo64) jinja2 : 2.10.3 IPython : 7.10.1 pandas_datareader: None bs4 : None bottleneck : None fastparquet : None gcsfs : None lxml.etree : None matplotlib : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pytables : None s3fs : 0.4.0 scipy : 1.3.3 sqlalchemy : 1.3.11 tables : None xarray : None xlrd : None xlwt : None xlsxwriter : None

[Liam3851]

@autocyz How quickly did you read the file after you changed it? S3 does not guarantee read-after-update consistency (see https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel). If you let less than 2-3 minutes go by between your update and running get_pid() you might have gotten a stale copy.

[autocyz]

@Liam3851 If I use code1, it never find the s3 change, but code2 can. I think it's not about s3 update consistency, it's about s3fs memory cache.

[TomAugspurger]

Do you think it's an s3fs issue then? It does cache file listings, and the instances of S3FileSystem are cached. You could try S3FS.clear_instance_cache.

[autocyz]

pd.read_csv default memory_map seem not to work, if I explicitly set memory_map to False, it work