Delay fetching Exchange item data from Graph API until it's clear kopia will be uploading the item

[ashmrtn]

Kopia-assisted incrementals allows us to skip uploading items that kopia already has to kopia again. However, the logic for GraphConnector still fetches all item data. This data fetch eats into the number of requests that graph API allows us

Instead of always fetching item data, we should only fetch the data if kopia requests it

[ashmrtn]

also related to

• 1702

  as it requires knowing when kopia actually needs data

[ashmrtn]

thinking about this more, it also has implications for how backup details for these items are handled. A plan for that should be pinned down prior to starting work on this

[vkamra]

Related also to #3007

[vkamra]

This (along with #3007) will help with scenarios such as:

• Delta token expired
• Delta-based backups cannot be leveraged because the mailbox is full

[ashmrtn]

Now that we added support for sourcing details entries from assist bases, the requirements for enabling kopia assisted incrementals have become a bit easier to meet. Currently the requirements are:

• complete ItemInfo is materialized if the item data is read by kopia (i.e. not required otherwise)
• item mod time is available
• item mod time matches mod time in ItemInfo if ItemInfo is materialized
• item name is available

Since we merge assist base details, we need only know the mod time and item name when first telling kopia about the item. We can put off materializing the ItemInfo until we know kopia read the file data and it wasn't cached

However, this leaves us with a couple new questions (numbered for ease of reference):

1. how do we handle errors when trying to fetch item data?
2. how do we handle items that are deleted between the time we enumerated them and the time we try to fetch their data?

For the first case, we can just return an error when opening the lazy reader. This will be caught by kopia and handled similar to an error fetching OneDrive/SharePoint data

For the second case, we need to be a little more careful (and this needs validated through manual testing). We should probably do three things:

• return a reader that has no data (i.e. create an empty file in kopia). We need to return some non-error because we don't want to fail the backup (by returning an error) but we've also already told kopia about the item (so it could see if it was cached) so we need to give it something
• return a sentinel error from Info() (call to get ItemInfo)
• check for the sentinel error when adding the item to the details builder. If we have the sentinel then don't add the details entry. This will make it so the item isn't searchable by users in the final backup

This should be sufficient because (in theory, needs validation in practice) the next backup should get a delta result saying the item was deleted. That delta result will cause us to exclude the item in kopia. The details entry for the (now excluded) item shouldn't exist anyway so the item will disappear completely from the new backup

Although having to check for sentinel errors in the kopia code is a bit messy, it does allow us to avoid having details entries that don't have data to back them

The error check can also be leveraged down the road to make sure the details are properly initialized when we return them (i.e. return an error if details was requested but the item data was never read)

If we go this route, the below help setup the error return value from the Info call

• 4174

• 4183

[ashmrtn]

re-opening this since we want to disable the feature and get more test coverage on it

[ashmrtn]

un-assigning for the moment since there's additional work that needs to be done to get this enabled