Intezer-Analyze short-circuit download

[ed4wg]

Is your feature request related to a problem? Please describe. Intezer counts download-file actions against a user's quota. See: https://support.intezer.com/hc/en-us/articles/360021366619-How-is-Your-Analysis-Quota-Calculated

Describe the solution you'd like To prevent un-necessary API hits against Intezer and depletion of the user's quota, I propose that when the assemblyline intezer service is processing sub-analyses, that it look in the AL holdings before trying to download the file from Intezer.

Describe alternatives you've considered Setting the config download_subfiles=False also prevents the quota from being used this way, but further analysis of the extracted files does not happen in this case.

[ed4wg]

I was looking at this a bit and wondering if there is an existing mechanism for both privileged and un-privileged services to access files in storage for download that is already accessible to analytic services? I did not see an existing way.

[gdesmar]

Would you know, for your Intezer use-case, if you have the subfiles by hash? Allowing a service to request another file is on our roadmap. For the moment, we are looking at adding requesting by hash (sha256, sha1, md5), or by display name based on the names in the current submission.

I believe Intezer is running as a privileged service, then if you have the sha256, it would be possible to do it today with the following:

from assemblyline.common import forge
fs = forge.get_filestore()
subfile_hash = <insert sha256>
fs.download(subfile_hash, os.path.join(self.working_directory, subfile_hash))

[ed4wg]

Would you know, for your Intezer use-case, if you have the subfiles by hash?

Yes, the service is attempting to download the sub-files from interzer by sha256. See: https://github.com/ed4wg/assemblyline-service-intezer/blob/6b3703033243d3baf6962c362247ce3d4b6a3fb7/intezer.py#L974

We don't run the service as privileged today, but could change that to make this solution work.