groboclown
commented
5 years ago Here's the cases of fstat being used by the plugin:
- getFiles OpenInDefaultChangelist. This searchesnthe entire client workspace, but with heavy option restrictions (-Olhp -Rc -Ro -e default), with a maximum file result count. The maximum count is a property in the plugin settings - maximum changelist results.
- getFileDetailsForOpenedSpecs. Fetches for information on the files open in one or more changelist (-Olhp), with a restricted maximum result count.
- getFileDetails. Called only for a single file.
- getFilesDetails. This looks like the culprit. It is being used exactly how you claim.
getFileDetails is used to find:
- Location information for a single committed file. This is enforced by setting maximum result count to 1.
- Find the current revision (have) of a file when performing a diff. This is restricted to 1 file.
- Calculating the current revision number when finding the history of a file. Restricted to 1 file.
I found a bug in getFilesDetails, where it wasn't respecting the requested maximum retrieve count. I don't think that is the soyrce of the issue, though, because in each instance, the query is made without a wildcard.
I'll keep digging and reportback with what I find.
wulfmayn
I'm the Perforce Senior Technical Lead with a large games company and on one of our teams we have a large project with approximately 500,000+ files in it - we've noticed that the plug-in is doing:
p4 fstat filesystem_path/...#have
and when it is referencing files high up our chain, we believe it's causing a memory leak over time each time it's called as the process is eventually consuming all available system memory and causing the P4 Server to stop responding. It's not due to one initiation of the command as Perforce has memory consumption protections per command to stop it if it detects it will consume too much memory.
Is there a better way of doing this? Polling with p4 fstat's against the local filesystem as opposed to the depot or even workspace path (if using streams), is extremely abusive and I'm not sure what is gained by doing a full recursive p4 fstat on the filesystem on every command that's can't be gained by calling more explicit paths that the user is actually viewing, or by using the depot or workspace file path spec as opposed to the local filesystem. The issue with the local filesystem is that has to scan and checksum every file it encounters, even one's outside source control.
Understanding why the plug-in needs to do this would be very helpful.
It's causing us significant instability for one of our triple A mobile titles so would be appreciated if we can work to sort this out - otherwise we have to pull the plug on this plug-in and the team would find that impactful to their workflows as they love this plug-in.