Too many close() syscall when collecting data

In my environment, about 1000000 close() syscalls produced by my own collector every 10 seconds. Almost takes 300ms when doing close FDs. After analysis, I find the following reasons.

It will close FDs(from 3 to MAXFD ) before launching a subprocess when using close_fds=True parameter in subprocess.Popen. https://github.com/OpenTSDB/tcollector/blob/master/tcollector.py#L1372 /usr/lib/python2.7/subprocess.py

    def _close_fds(self, but):
        if hasattr(os, 'closerange'):
            os.closerange(3, but)
            os.closerange(but + 1, MAXFD)
        else:
            for i in xrange(3, MAXFD):
                if i == but:
                    continue
                try:
                    os.close(i)
                except:
                    pass

In my env, MAXFD=1000000

Also, after review the history patch, I find the patch which setclose_fds from False to True. https://github.com/OpenTSDB/tcollector/commit/7b5659db0dfbc3915cf16f90e3fbf92e1deec1da#diff-50bd42e8d38bbdf7c24f69ae5e25165f @tsuna

Any good idea to solve to this?

OpenTSDB / tcollector

Too many close() syscall when collecting data #417