search

ligershark / side-waffle

A collection of Item- and Project Templates for Visual Studio
Other
656 stars 205 forks source link

Add Telemetry (tracking usage) #178

Open sayedihashimi opened 10 years ago

sayedihashimi commented 10 years ago

Currently for usage the only data that we have is the download count from the VS gallery. We currently have no clue which templates are getting used. It would be good if we could track this info somehow. I am not sure what all our available options are here. I'm also not sure if anyone would mind if we added this. Having this data would help us prioritize new features as well as understand in which areas we should continue to invest.

Proposal

Here are my initial thoughts.

We should be able to track usage for SideWaffle as well as other template packs created by TemplateBuilder. One way that I can think of to enable this is to create a custom wizard that will hit a tracking URL and pass query parameters with the following info.

  1. Package ID for the VSIX containing the template
  2. TemplateID if in the .vstemplate
  3. Template Name
  4. Template type (project template or item template)

We can update the TemplateBuilder build process to inject the custom wizard when the vsix is built. For those who are interested in "opting out" we can add an MSBuild parameter which can be used to disable the functionality (it will be on by default).

Questions

  1. Any concerns with tracking this data? If so is the opt out proposal I have good enough?
  2. Is there any (hopefully free) services available that can help us track/report on this?

    How to implement

We will need to do the following to get this implemented.

codewithtyler commented 10 years ago

So just to clarify, when a user chooses to a template you want to pass the info to a web server telling it what template they chose?

sayedihashimi commented 10 years ago

@RandomlyKnighted yes that's the proposal.

sayedihashimi commented 9 years ago

Here is a blog post that has some details about how to add this http://igrali.com/2013/09/04/add-google-analytics-to-your-windows-phone-and-windows-store-apps-in-3-easy-steps/.

sayedihashimi commented 9 years ago

OK so it seems like there is no concerns with us adding telemetry to SideWaffle/TemplateBuilder. I'll start working on this soon. Currently we have no idea what templates are being used. It would be very helpful to have an understanding of which templates are getting used, and which are not. We will have an option to disable the telemetry as well.

My initial thoughts are to use google analytics.

codewithtyler commented 9 years ago

In regards to Step 1, it looks like the "SideWaffle Item Template" item template does not add a TemplateId when it adds the .vstemplate files.

Were you able to find out how to add a custom wizard to project/item templates by default?

sayedihashimi commented 9 years ago

@RandomlyKnighted

it looks like the "SideWaffle Item Template" item template does not add a TemplateId

I just updated them in b6548b29c6d72a207d879575090599aca7ac4630, item and project template template now both have TemplateId.

Were you able to find out how to add a custom wizard to project/item templates by default?

All we have to do is to update the _project.vstemplate.xml or *.vstemplate file to have an additional WizardExtension element (example). In this case instead of adding one by default we just have to update the files in this repo to use this new custom wizard. And when we receive new templates make sure they have the custom wizard in the files.

We also will need to update every template file to have a TemplateId. We can do this with PowerShell pretty easily.

codewithtyler commented 9 years ago

Correct me if I'm wrong, but I'm understanding this as we can add something like the following to each vstemplate file

  <WizardExtension>
    <Assembly>LigerShark.Templates, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null</Assembly>
    <FullClassName>LigerShark.Templates.SideWaffleTelemetryWizard</FullClassName>
  </WizardExtension>

Where the FullClassName element points at a class in some project that we create. Is that right?

sayedihashimi commented 9 years ago

Correct

codewithtyler commented 9 years ago

Does it have to be a specific kind of project? (i.e. Console Application, Class Library, etc.)

codewithtyler commented 9 years ago

@sayedihashimi In the HTML5 UP! Wizard that you created I noticed that that each function contained an argument for a project. How is the project being passed to each of the functions? The reason I ask is because I'm wanting to set up the telemetry wizard so that the TemplateID is the argument.

codewithtyler commented 9 years ago

I've created a function that uses the Google Analytics SDK to track which template is created. There are still three things left that need to be done.

Whenever you go to store the information just reading the Template ID isn't going tell you which template that is unless you look it up. So do we want to store more than just the Template ID in order to make reading the information easier?

sayedihashimi commented 9 years ago

@RandomlyKnighted I think we should send TemplateID and Name. I'll look into how to get those values, hopefully tonight.

sayedihashimi commented 9 years ago

@RandomlyKnighted I looked around to see if I could figure out how to get the TemplateID/Name inside a custom wizard and didn't make much progress. I've emailed a few people who should be able to give me more info. I'll update this when they reply.

sayedihashimi commented 9 years ago

@RandomlyKnighted OK I've heard back. I was wrong in my approach to solve this, evidently you cannot access the .vstemplate metadata from a custom wizard. Instead in the .vstemplate file we can define a Custom parameter for the ID and Name. Then when the custom wizard is invoked we can use that data to track the usage.

I think we should create the following custom params for each template;

codewithtyler commented 9 years ago

So we replace the TemplateID with the Custom Parameters? If that's the case, then we could probably do that with PowerShell.

sayedihashimi commented 9 years ago

There's no benefit in removing existing IDs. We can just create a script to add the new elements, and the guids can/should be different.

codewithtyler commented 9 years ago

If we create a new guid for this then the old TemplateID would become an identifier for Visual Studio and this one would be an identifier for us, correct?

sayedihashimi commented 9 years ago

Yes. The issue with copying it is that it will get out of sync and there's no benefit to syncing them.

codewithtyler commented 9 years ago

Makes sense. So I guess our next step is to update all the vstemplate files with the TemplateExtension and Custom Parameters. I'm still pretty new to PowerShell so I could use some help with this part.

sayedihashimi commented 9 years ago

See my script to add template id https://github.com/ligershark/side-waffle/blob/master/add-templateid.ps1 that should get you started

codewithtyler commented 9 years ago

Thanks, I was able to get my script to work. Now, I just need to get the TemplateName and TemplateID from the wizard parameters and pass it to my function then I'll be ready to upload a VSIX to Dropbox so we can test it.

sayedihashimi commented 9 years ago

@RandomlyKnighted I looked at the wizard code and the values are not coming in via customParams but instead they are in replacementsDictionary. See the image below.

replacements

codewithtyler commented 9 years ago

How were you able to step through the code like that? Anytime I try to debug it using the Experimental Instance it just skips my breakpoint?

Update: I was able to get the values from replacementsDictionary by using $TemplateName$. However, in the MSDN documentation it shows that you use $TemplateName to get the value. Is that the way it used to be? If so, then it would seem that the documentation is outdated.

sayedihashimi commented 9 years ago

@RandomlyKnighted yeah I see that too not sure what causes it. What I did was add System.Diagnostics.Debugger.Launch(); to the line that I wanted to debug and then F5 on that.

Not sure about the docs.

codewithtyler commented 8 years ago

@sayedihashimi did we ever update SideWaffle so that new project and item templates automatically add the Wizard?

sayedihashimi commented 8 years ago

I thought that I added that a while back, are you not seeing it in the result?

codewithtyler commented 8 years ago

Where would it have been added at?

codewithtyler commented 8 years ago

@sayedihashimi Ok, I've done some digging and I know why it was throwing the error

The given key was not present in the dictionary.

It appears that the TemplateName and TemplateID are not being added to the replacementsDictionary. I'm still trying to figure out why they are not being added.

codewithtyler commented 8 years ago

@sayedihashimi When you get some time could you take a look at this and make sure I'm not missing anything. I compared the xUnit project template to the Caliburn.Micro Project Template. The wizard nodes and custom parameter nodes for both projects look identical. Yet, the Caliburn.Micro project goes through the wizard with no problem. When I create a xUnit project it gives me the error mentioned above, which you saw a couple weeks ago when we did our Skype call. I don't see why it's not adding the TemplateName and TemplateID to the replacementsDictionary for some templates.

sayedihashimi commented 8 years ago

@RandomlyKnighted I figured out the exception. The issue is that TemplateBuilder is removing the CustomParameters entry. I filed a new bug for it at https://github.com/ligershark/side-waffle/issues/321. I'll take care of that. I'll also wrap the RunStarted code in a try/catch so that the exception is never raised.

codewithtyler commented 8 years ago

Sounds good. Let me know if there's anything else that I can do in order to get this new version released.

sayedihashimi commented 8 years ago

@RandomlyKnighted I'll take care of this. It's kind of involved since it requires changes to TemplateBuilder itself. Once I make the changes it will be good for you to take a look at the changes I made so that you're familiar with them.

sayedihashimi commented 8 years ago

@RandomlyKnighted OK I had to update TemplateBuilder with the following https://github.com/ligershark/template-builder/commit/4c9c4521f917861527e4c5fe150caccd59376b97 and then I updated SideWaffle to get the latest TemplateBuilder nuget package with https://github.com/ligershark/side-waffle/commit/ac224da412205133914c90839398005d6be73e79.

That should fix the telemetry issue. The build is still going, when it's complete I'll test the .vsix quickly and then kick off a release from here.

After this I'll work on getting your website changes included. I may need to update my webserver for that so it may take some time. Anything pressing besides that which I missed?

sayedihashimi commented 8 years ago

Fixed now and I just kicked off a publish https://ci.appveyor.com/project/sayedihashimi/side-waffle/build/1.19.325

codewithtyler commented 8 years ago

From what I can tell there are only a couple of things left to do in order to complete this.

codewithtyler commented 8 years ago

In issue #370 it as mentioned that users have to disable the Google Analytics Wizard in order to create their template. @sayedihashimi is there a way that we can disable the Wizard after it reports the template to us?

henrik-jensen commented 8 years ago

This is a follow up on some comment of mine in issue #370 I felt is more appropriate here:

Though the initial proposal mentions some possibility of disabling the functionality, this part seems to have been neglected in the progress of this project.

No matter how benevolent shouldn't it be mention very upfront, both on the website, Visual Studio Gallery and in the wiki1, that sidewaffle is, and why it is, tracking the usage of its users and describe an easy method to disable this tracking?

As I argued in #370 I would prefer it to be some kind of opt-in, though I would understand if this could be awkward to implement in this scenario.

1. As I understand from #370 @RandomlyKnighted is in the process of adding some documentation of this to the wiki.

codewithtyler commented 8 years ago

@henrik-jensen Disabling the functionality was never intentionally neglected. We were waiting to make sure the reporting worked as we expected it to before moving forward to disabling it as a feature.

At the moment there is no way to disable it. That will have to be added in. The only thing you can do at this time is comment out the Wizard when you create your own templates.

codewithtyler commented 8 years ago

I've just looked over the code and there's some good news. It looks like we were already checking to see if the user has opted-out or not. We just never finished adding the user setting to disable it. I'm working on adding that now.