Query on LogConfiguration

suryatn commented 3 years ago

We used Aria SDK's LogConfiguration to set the custom values of collectorUrl, cacheFileName, enablePauseOnBackground using the code snippet

LogConfiguration logConfig = new LogConfiguration();
if (!StringUtils.isNullOrEmptyOrWhitespace(getCollectorUrl())) {
    logConfig.setCollectorUrl(getCollectorUrl());
}
logConfig.setCacheFileName(getCacheName());
logConfig.enablePauseOnBackground(false);

LogManager.appStart(application, mAriaTenantToken, logConfig);

Sample Values
collectorUrl (String)- "https://mobile.pipe.aria.microsoft.com/Collector/3.0/"
cacheFileName (String) - "AriaStorage.db"

How can we achieve the same in 1DS SDK. I am trying via below snippet

ILogConfiguration logConfig = LogManager.logConfigurationFactory();
if (!StringUtils.isNullOrEmptyOrWhitespace(getCollectorUrl())) {
    logConfig.set(LogConfigurationKey.CFG_STR_COLLECTOR_URL, getCollectorUrl());
}
logConfig.set(LogConfigurationKey.CFG_STR_CACHE_FILE_PATH, getCacheName());

LogManager.initialize(tenantToken, logConfig);

Please confirm if CFG_STR_COLLECTOR_URL corresponds to AriaSDK's collectorUrl and CFG_STR_CACHE_FILE_PATH corresponds to cacheFileName. These 2 LogConfigurationKeys are the closest match that I found. Also there is no option to set enablePauseOnBackground in 1DS? What are the possible values accepted for CFG_STR_COLLECTOR_URL and CFG_STR_CACHE_FILE_PATH?

@larvacea @mkoscumb @maxgolov @sid-dahiya

lalitb commented 3 years ago

Please confirm if CFG_STR_COLLECTOR_URL corresponds to AriaSDK's collectorUrl and CFG_STR_CACHE_FILE_PATH corresponds to cacheFileName.

That's correct

Also there is no option to set enablePauseOnBackground in 1DS.

In 1DS, you can use LogManager::PauseTransmission() / LogManager::ResumeTransmission() to respectively pause and resume the transmission of events to data collector. When paused, these events will be queued in client side in cache (CFG_STR_CACHE_FILE_PATH ) and /or memory.

What are the possible values accepted for CFG_STR_COLLECTOR_URL and CFG_STR_CACHE_FILE_PATH

CFG_STR_COLLECTOR_URL should be valid URI as defined by RFC 3986. CFG_STR_CACHE_FILE_PATH should be valid file-name, with set of valid characters defined by the file-system where this file will get created. Just to add, the file location should be writable by the process.

@maxgolov and others can add more details/clarifications here if I missed.

suryatn commented 3 years ago

@lalitb sure, thanks. Can you also let me know the significance of collector url? I am assuming different regions have different collector end points where in the data gets sent to different collector end points accordingly.

For cache name, the values which we use for currently are AriaStorage.db, army_aria.db, gcc_high_aria.db depending on whether it is public/gcc. The naming in 1DS is CFG_STR_CACHE_FILE_PATH, which is actually just a file name and not path correct? These cache files will be created on the client side and not the server correct?

@lalitb @maxgolov I also see that this property enableAutoUserSession is missing in 1DS. It's is present in Aria SDK with default value as false. Is there any equivalent in 1DS?

lalitb commented 3 years ago

Can you also let me know the significance of collector url? I am assuming different regions have different collector end points where in the data gets sent to different collector end points accordingly.

Region specific Aria URI's are defined in ILogConfiguration.hpp as below: COLLECTOR_URL_GERMANY = "https://emea.events.data.microsoft.com/OneCollector/1.0/"; COLLECTOR_URL_AUSTRALIA = "https://apac.events.data.microsoft.com/OneCollector/1.0/" COLLECTOR_URL_JAPAN = "https://apac.events.data.microsoft.com/OneCollector/1.0/"; COLLECTOR_URL_EUROPE = "https://emea.events.data.microsoft.com/OneCollector/1.0/";

If you want to send Logs across multiple regions simultaneously, you can create multiple log managers as in example .

The naming in 1DS is CFG_STR_CACHE_FILE_PATH, which is actually just a file name and not path correct?

Correct. The default file path is based on the platform, and is defined here.

These cache files will be created on the client side and not the server correct?

Correct

I also see that this property enableAutoUserSession is missing in 1DS. It's is present in Aria SDK with default value as false. Is there any equivalent in 1DS?

Log Sessions are enabled by default, and session file gets created at same location as event cache file. As of now, this is not enabled for UTC upload, but this will change soon.

suryatn commented 3 years ago

@lalitb thanks for the clarification. Will ping here incase I have any further queries on this.

suryatn commented 3 years ago

@lalitb Also just to confirm, we need to use the collector urls mentioned in the ILogConfiguration incase we want to log data to specific regions correct? For Aria the values used are as follows

public static final String COLLECTOR_URL_GERMANY = "https://de.pipe.aria.microsoft.com/Collector/3.0/";
public static final String COLLECTOR_URL_AUSTRALIA = "https://au.pipe.aria.microsoft.com/Collector/3.0/";
public static final String COLLECTOR_URL_JAPAN = "https://jp.pipe.aria.microsoft.com/Collector/3.0/";
public static final String COLLECTOR_URL_EUROPE = "https://eu.pipe.aria.microsoft.com/Collector/3.0/";

I am assuming these endpoints will not work for 1DS and we have to change to those mentioned in the ILogConfiguration.hpp of 1DS for them to work.

For prod case, default collector url (COLLECTOR_URL_PROD = "https://self.events.data.microsoft.com/OneCollector/1.0/") will be automatically picked up correct?

lalitb commented 3 years ago

I am assuming these endpoints will not work for 1DS and we have to change to those mentioned in the ILogConfiguration.hpp of 1DS for them to work.

I am not aware of these regional URL's from Aria SDK. Adding @maxgolov if he has the answer

For prod case, default collector url (COLLECTOR_URL_PROD = "https://self.events.data.microsoft.com/OneCollector/1.0/") will be automatically picked up correct?

That's correct

suryatn commented 3 years ago

I am assuming these endpoints will not work for 1DS and we have to change to those mentioned in the ILogConfiguration.hpp of 1DS for them to work.

I am not aware of these regional URL's from Aria SDK. Adding @maxgolov if he has the answer

For prod case, default collector url (COLLECTOR_URL_PROD = "https://self.events.data.microsoft.com/OneCollector/1.0/") will be automatically picked up correct?

That's correct

@maxgolov can you please confirm on the first point?

mkoscumb commented 3 years ago

Do not use the old aria endpoints with the new 1DS SDK, the protocols are different and data will be rejected.

Similarly, do not use the new 1DS SDK endpoints with the old Aria SDK for the same reason.

mkoscumb commented 3 years ago

As to enablePauseOnBackground, we recommend calling LogManager::PauseTransmission() and LogManager::Flush() when your app is Suspended, then calling LogManager::ResumeTransmission() on resume from background.

The SDK does not have any knowledge of whether the calling application is in the background or foreground, nor whether or not the underlying OS has any support for such a metaphor.

larvacea commented 3 years ago

On Android, the database file name is set by the CFG_STR_CACHE_FILE_PATH value, as you thought. There are two persistent database implementations available in the Android SDK, so these files end up in slightly different places (and with different contents) depending on the implementation:

For the Android Room, they end up in /data/data/com.microsoft.applications.events.maesdktest/databases/
For SQLite, they end up in /data/data/com.microsoft.applications.events.maesdktest/cache/

In each case com.microsoft.applications.events.maesdktest is the application's id/packageName, i.e., the special file path for this particular application. So now you know. In either case, they are in internal, application-specific storage.

suryatn commented 3 years ago

@lalitb Also just to confirm, we need to use the collector urls mentioned in the ILogConfiguration incase we want to log data to specific regions correct? For Aria the values used are as follows
public static final String COLLECTOR_URL_GERMANY = "https://de.pipe.aria.microsoft.com/Collector/3.0/";
public static final String COLLECTOR_URL_AUSTRALIA = "https://au.pipe.aria.microsoft.com/Collector/3.0/";
public static final String COLLECTOR_URL_JAPAN = "https://jp.pipe.aria.microsoft.com/Collector/3.0/";
public static final String COLLECTOR_URL_EUROPE = "https://eu.pipe.aria.microsoft.com/Collector/3.0/";
I am assuming these endpoints will not work for 1DS and we have to change to those mentioned in the ILogConfiguration.hpp of 1DS for them to work.

For prod case, default collector url (COLLECTOR_URL_PROD = "https://self.events.data.microsoft.com/OneCollector/1.0/") will be automatically picked up correct?

@lalitb @maxgolov @mkoscumb @sid-dahiya @larvacea What should be the 1DS collector url endpoints for GCCH and DOD scenario For Aria we use

public static final String COLLECTOR_URL_USGOV_DOD = "https://pf.pipe.aria.microsoft.com/Collector/3.0"; // For DOD
public static final String COLLECTOR_URL_USGOV_DOJ = "https://tb.pipe.aria.microsoft.com/Collector/3.0"; // For GCCH and GCC

Are there similar endpoints for those in 1DS? I cannot see them in ILogConfiguration.hpp. We would be needing them as well.

maxgolov commented 3 years ago

@suryatn - for non-public cloud end-points, please take this offline with Andrew Faust - discuss it by email, please. @aef123

suryatn commented 3 years ago

On Android, the database file name is set by the CFG_STR_CACHE_FILE_PATH value, as you thought. There are two persistent database implementations available in the Android SDK, so these files end up in slightly different places (and with different contents) depending on the implementation:

For the Android Room, they end up in /data/data/com.microsoft.applications.events.maesdktest/databases/

For SQLite, they end up in /data/data/com.microsoft.applications.events.maesdktest/cache/

In each case com.microsoft.applications.events.maesdktest is the application's id/packageName, i.e., the special file path for this particular application. So now you know. In either case, they are in internal, application-specific storage.

@larvacea @maxgolov @mkoscumb @sid-dahiya @lalitb

I have couple of more queries related to LogConfiguration's CFG_STR_CACHE_FILE_PATH,

What is the size limit for the cache file? Is it same as that of Aria SDK?
What happens when we reach the size limit? We would want to preserve High and Intermediate priority events. Is it possible to not drop these events? How different is it from that of Aria SDK?

Other queries related to the 1DS logger initialisation part

How much time does it take to initialise the 1DS Logger i.e LogManager.initialize() compared to that of Aria SDK?
Is the initialisation part synchronous in case of 1DS?

maxgolov commented 3 years ago

Size is configurable. By default same as Aria. Configuration parameter is described here: https://github.com/microsoft/cpp_client_telemetry/blob/e99ec198169a571361bc74ebedaf3d56696ca348/lib/include/public/ILogConfiguration.hpp#L145

When you reach the size limit, default behavior is not truncate while the app is running. But truncate when the app restarts. High and Immediate preserved, lower priority evicted first. This is no different than Aria SDK.

How much time does it take to initialise the 1DS Logger i.e LogManager.initialize() compared to that of Aria SDK?

There could be no answer to that question. You have to profile. Typically milliseconds to 50ms to 100ms, it depends on hardware. Initialization is synchronous.

suryatn commented 3 years ago

@maxgolov @larvacea is default cache file size incase of 1DS is 3MB or 16MB? Can you please confirm? https://github.com/microsoft/cpp_client_telemetry/blob/e99ec198169a571361bc74ebedaf3d56696ca348/docs/Offline-storage-settings.md I have seen some where in go wrapper to be 16MB.

I think for Aria it is 10MB as per the below line in Aria SDK code

private int cacheFileSizeLimitInBytes = 10485760;

We would want to keep it similar to Aria SDK config values. Also can you point to the truncate logic when the limit is reached?

maxgolov commented 3 years ago

as per the below line in Aria SDK code

We do not support Aria SDK. So, unfortunately, cannot comment on what may or may not be set in Aria SDK.

Go wrapper can set whatever value that they see fit for their environment.

You can also set whatever value that works for your product. Check what's in Aria SDK, then pass the same values to ILogConfiguration. Please review ILogConfiguration.hpp ( https://github.com/microsoft/cpp_client_telemetry/blob/master/lib/include/public/ILogConfiguration.hpp ) for the list of available parameters. Each parameter has a description next to it in code comment. Please check the code comments. You can also use Doxygen to generate a cross-reference for public headers. We support Doxygen.

maxgolov commented 3 years ago

Default parameters for 1DS C++ SDK are here: https://github.com/microsoft/cpp_client_telemetry/blob/master/lib/config/RuntimeConfig_Default.hpp

suryatn commented 3 years ago

@maxgolov @larvacea @mkoscumb @sid-dahiya one more question regarding getting 1DS's LogMananger.

What is the significance of these methods LogManager.initialize() and LogManager.getLogger()? Will the events be logged with the ILogger instance obtained through these methods and if yes, which tenant/DB will the logger be logging events if no tenant token is passed?

Also what does a source passed in getLogger(tenantToken, source) signify, what is the use of it?

maxgolov commented 3 years ago

What is the significance of these methods LogManager.initialize() and LogManager.getLogger()

My understanding is that your application has been already using Aria SDK. These are the same identical concepts as in Aria SDK. GetLogger - obtains a Logger, Initialize - starts-up the entire telemetry system. This question is also somewhat unrelated to ILogConfiguration. Source -- allows to have separate per-module loggers sending data to the same destination storage DB, but with its own semantic context variables.

https://1dsdocs.azurewebsites.net/api/cppSDK/classLogManager.html https://1dsdocs.azurewebsites.net/api/cppSDK/classILogger.html

which tenant/DB will the logger be logging events if no tenant token is passed?

GetLogger() returns the default primary logger associated with primary token you'd initially pass to Initialize method.

suryatn commented 3 years ago

@maxgolov sure, what happens if we initialise 1DS by LogManager.initialize() with out tenant token and do LogMananger.getLogger(). Which Logger we would be getting and where will the data be logged? I see that we use default token, will we be able to access data in that case?

maxgolov commented 3 years ago

with out tenant token

Sorry, you meant without or with our ?

If you initialize without a token and don't supply one in ILogConfiguration, nothing good is gonna happen. Since you are not telling the system where to send the data.

If you initialize with your token T1, then use GetLogger() - expected behavior is that you obtain a logger associated with primary token T1 you'd be passing to LogManager::Initialize(T1).

I would suggest you to run a few samples on Windows, using Fiddler inspector on Windows. That is the fastest way to learn more about how SDK operates.

suryatn commented 3 years ago

Sure, thanks for the confirmation @maxgolov

microsoft / cpp_client_telemetry

Query on LogConfiguration #614