Can we use the PublicBody#notes attribute to improve Adwords traffic?

[garethrees]

Q: Can we use the PublicBody#notes attribute to improve Adwords traffic?

Does using the notes fields on authorities to write out a description of the kinds of information you might ask them for e.g. immigration processing times, and matching that to adwords increase the traffic to the site? Or, for a university, success rates based on previous applicants' qualifications

[Explain how you will measure this]

Conversion rate from ad matching notes description

[What do we need to do to be able to take the measurements?]

• [x] Add notes fields for the authorities containing the kinds of requests that these authorities get in the UK but that aren't being made in the country
• [x] Set up a conversion on Google Adwords
• [x] Add tracking code to RTK website
• [x] Measure traffic/conversions to authority from the tailored Adwords

[lizconlan]

Note to self: look for data from RightToKnow (Aus)

[garethrees]

18 Mar 2016 Added notes to university authorities indicating that they would hold admission statistics and policy.

[lizconlan]

need to ask OpenAustralia to set up targeted ads for this

[Gemmamysoc]

Next step is to see if more people have landed on university pages than before the notes were added to description field.

[Gemmamysoc]

Then, based on the above, we could see if we think setting up targetted Adwords ads would help

[Gemmamysoc]

Method:

On Google Analytics, I set the timeline to compare 18 December 2015 - 18 March 2016 to 18 March 2016 - 13 June 2016. 18 March was the date that Louise added descriptive text to all 21 university bodies.

I then went to 'Behaviour' - 'Site content' - 'Landing pages' and searched for the body pages (e.g. /body/deakin_university) for each of the 21 universities that Louise added descriptive text to.

Results:

7 out of 21 (33%) universities had an increase in sessions on their body pages after the descriptions were added.

14 (66%) universities had the same amount of sessions on their body pages than before (usually 0).

Only one university (4%) had more sessions on their body page before the description was added.

Results for each university where session increased:

100% increase:

• Charles Sturt University - had 1 more session on body page after description was added
• La Trobe University - had 1 more session on body page after description was added
• Southern Cross University - had 1 more session on body page after description was added
• Swinburne University of Technology - had 1 more session on body page after description was added
• University of Wollongong - had 1 more session on body page after description was added

300% increase:

• University of New South Wales - had 3 more sessions on body page after description was added

400% increase:

• University of Sydney - had 4 more sessions on body page after description was added

University that had more sessions on body page before text was added:

University of Melbourne - had 1 more session on body page before description was added

[Gemmamysoc]

Further to the above, I've discovered that:

• University of Sydney (had 4 more sessions on body page after description was added) - the source of all 4 sessions came from Facebook, not Google :(
• University of New South Wales (had 3 more session on body page after description was added): 2 of these came Direct and 1 came from Google. Keyword not set/visible.

Universities that had one more session on body page after description was added:

• Charles Sturt University: Source was Google
• La Trobe: Source was Google
• Southern Cross: Source was Google. Keyword: 'southerncrossuniversity'. Bounce rate was 100%.
• Swinburne University of Technology: Source was Direct.
• Wollongong: Source was Google

Out of the 12 new sessions on pages since description was added, 5 (42%) came from Google.

[Gemmamysoc]

In the period after the text descriptions were added, only one new request was made - to the Australian National University. All of the other universities have 0 requests. Only the Australian National University has had requests - 3 in total.

[garethrees]

So on the surface it looks like this is encouraging, but if we're talking about "1 more session" as a 100% rise then I'm not massively convinced that we can claim this is a definite improvement.

I think the targeted Adwords are more likely to have a significant impact here.

What do we need to do to push this forward?

[garethrees]

Discuss in Sheffield

[Gemmamysoc]

Next step: Think about what advert text would be appropriate

[garethrees]

• Look at requests made to WDTK universities
• Pick out common themes that people are asking (admissions statistics, course withdrawal statistics, cost of X, expenses of X, grade thresholds, etc)
• Make ads for these, for each AUS university ("Use FOI to find X university grade thresholds")
• Present to RTK team

[MyfanwyNixon]

After a chat with Gemma, I'm going to make a start on this myself and possibly hand back over if I reach a stage where that's practical.

[MyfanwyNixon]

Note: The scope seems to have widened since the initial addition of 'admissions statistics and policy' being added to the notes attribute.

I picked out ten common themes:

• Grade thresholds/boundaries
• Application (figures/ stats/numbers)
• Accounts/expenditure/expenses
• Admissions/entry (stats)
• Interviews (figures/percentages/statistics/scoring system/rates)
• Grades (A level*)/entry requirements of admitted students/entry grades)
• Library (costs/data/stats/income/donations/fines)
• animal testing/experiments/research
• exam results/data
• Staff salaries/wages/salary bands

Given the best practice of creating an AdGroup for each likely wording within each topic (eg one for 'university expenditure', one for 'university expenses'), we may end up with more AdGroups than initially envisaged -- especially as, for maximum efficacy, I think we'd also create a separate AdGroup within each topic for each university, thus allowing us to link directly to that uni's page. That's looking like over 400 AdGroups.

So, for the purposes of this experiment I think I am going to scale back:

• Only AdGroups/ads/keywords pertaining to Admission Policy and Statistics
• Only the top 3 universities in Australia, plus a generic set that covers all universities.

If this goes well, we can roll out the rest. Does that sound reasonable?

*Year 12 Cert is the equivalent; this also has several alternatives depending on the state.

[Gemmamysoc]

Thanks so much for your work on this Myf! Yes, your scaling back plan sounds like a good one to me. Wow - managing and improving Adwords really could be a full time job couldn't it! (I'm sure it is for some people :) )

[garethrees]

That's looking like over 400 AdGroups.

So, for the purposes of this experiment I think I am going to scale back:

• Only AdGroups/ads/keywords pertaining to Admission Policy and Statistics
• Only the top 3 universities in Australia, plus a generic set that covers all universities.

If this goes well, we can roll out the rest. Does that sound reasonable?

Yep, sounds great!

[MyfanwyNixon]

Here is the spreadsheet ready for the RTK team to upload: https://docs.google.com/spreadsheets/d/1HXvS5O0RGRHXNB_xzMnKqEO7VrPKsZJwd7Z8Z-2E3Jc/edit#gid=2011092382

As agreed, it contains Adgroups, Keywords and Ads for the top 3 universities (University of Melbourne, of Sydney and Australian National University) and for universities in general. The Keywords and Adgroups are all around the topics of admissions policies and stats.

Uploading the spreadsheet will create a new Campaign for the called RightToKnow Experiments, since I am guessing they will want to keep these Adgroups separate from any they already have set up (feel free to change to the existing RightToKnow Campaign if that's not the case).

I haven't set bids or filled in many other fields, which means they will all default to whatever is the current default in RTK's account.

Instructions:

• Download Adwords Editor (if they don't already have it) - this allows for bulk uploads to an Adwords account.
• Instructions for bulk uploading from a csv file are here.
• Hopefully no errors will occur but if they do, the error messages should help you identify what is wrong. Potentially this might be the character counts of Ad titles and text: I've tried to check that they are within limits but there's always potential for mistakes.

Improvements the RTK team could make

Once the Ads are uploaded, the RTK team may wish to:

• Use their own knowledge, and the Google suggested keywords tool, to add more keywords to each Adgroup
• Check that the language used is fitting for Australian users, since it was written by an English person!
• After the ads have run for a couple of weeks, check back to see which formats are getting the highest click through rates, and replicate these across other Adgroups.

Factors which may make this experiment imperfect

• I have linked to the RTK homepage for the generic university keywords, and to the university's page for the keywords mentioning a specific institution. These pages may be a drop-off point as it isn't especially obvious what to do next and the pages do not reflect the keyword. If we really wanted these Adwords to work I'd suggest a landing page such as the ones we have in place for TWFY and FMS, eg https://www.mysociety.org/wehelpyou/who-is-your-mp/
• Google will also downgrade these ads for the fact that the landing pages aren't massively specific to the keyword and the ad.

Additional thoughts

• Another experiment might point to existing requests rather than try to encourage users to make their own straight away. Such requests might display more clearly exactly how the site can be used to obtain what the user wants. If there are no such requests already, it would not be too difficult to make some within these limited subject areas.
• If the experiment is successful and we want to extend it, there is a full list of potential Adgroups, covering the 10 popular topics, here (tab 4): https://docs.google.com/spreadsheets/d/1dBnvxzLv1WD6ktRxW9qciShqMrONhk2GjaATQ2q68fs/edit#gid=754449678
• We may also want to replicate this for WDTK, since we do not currently run Ads pointing to our university pages.

[MyfanwyNixon]

Also: to measure the impact properly, RTK will want to have conversion tracking set up, measuring how many people click on an ad and go on to make a request. (They may already have it, as it is a condition of the higher tier Adwords grant.)

[MyfanwyNixon]

With impeccable timing, Google have introduced a new format for Ads: https://support.google.com/adwords/answer/7048854?hl=en-GB&authuser=0

Ads will now be slightly longer and have two header lines instead of one. Existing ads will continue to run, but after October 26 won't be editable apparently.

What this means for this experiment:

• It's possible that the newest version of Adwords Editor won't upload the old format, but I haven't tried
• Old style ads will be retained, just not editable, so we can still get data from them. Eventually they'll probably be phased out and the account owner encouraged, increasingly urgently, to rewrite in the new format, but typically Google spreads this sort of changeover over many months. -I could rewrite all the ads to fit the new format if we think it's worth it. It'd be quite annoying but give the ads better longevity - eg if you think there's going to be a need to compare like with like over different future time periods. The longer text would also allow for better messaging!

[garethrees]

I'd be inclined to go with what we have. End of October is 3 months away, so plenty of time for these to run and so something useful.

[Gemmamysoc]

I can contact RTK team about this :)

[kat]

Hi @MyfanwyNixon I had a go at this and got as far as the import instruction which failed, returning this error:"A required account column is missing". Have you seen that one before?

[kat]

@MyfanwyNixon I've sent and invitation so you can access to the adwords account & troubleshoot directly if that helps.

[MyfanwyNixon]

Hi all, sorry for the error! I'm still on holiday but if you'd like to have a go at troubleshooting this yourself, please take a look at https://support.google.com/adwords/editor/answer/30564?hl=en-GB&ref_topic=2986631 where you'll also see a template for a blank spreadsheet. It might help to compare the columns on that one with the one I provided.

If that doesn't help, my other thought is that the new format for Adwords (as mentioned above) might mean that the old spreadsheet is somehow invalid, which is annoying - but perhaps there's a new version somewhere if you poke around.

I'm afraid my knowledge of this isn't much greater than 'gah, it always errors, turn it over to a developer' so if neither of those things help, maybe seek help from Gareth or another more technically-inclined member of the team!

[MyfanwyNixon]

Hi all, Thanks to Henare for giving me access to the Adwords account.

As you'll see, there's now a Campaign in your Adwords called RightToKnow experiments. Please feel free to go in and lower/raise the budget or make any changes as you see fit. The ads are live, so with a bit of luck we'll be able to assess how well they are working within a few days.

Right now lots of them are showing as 'low search volumes' so it may well be that the keywords I've chosen aren't the best: now the structure is in place, though, there's scope to add or remove keywords too.

@kat - it turns out the solution to the ambiguous row types was here: http://nickbartlett.com/adwords-editor-tool-ambiguous-row-type/ I followed that advice and it worked!

[garethrees]

Looks like some of the ads have encoding issues:

[MyfanwyNixon]

Damn it! Shouldn't be too hard to go in and fix manually though - will have a look this morning.

[MyfanwyNixon]

OK, it looks like it was just that one ad - fixed now.

[MyfanwyNixon]

Hi all, Looking at the ads, there's a nice rate of clicks coming through on the generic Entry Requirements ad group (presumably seasonally relevant?).

The others are pretty slow and several have had 0 or low impressions and 0 clicks. It's early days yet, I suppose, and perhaps some of these will kick in later. In total there have been 919 clicks from the ads.

Entry requirements, entry grades and application statistics have very good click through rates (ie impressions divided by clicks) - up to 8.22% in the case of Application Statistics.

As I mentioned above, real insights are only going to be gained if conversion tracking is set up - then we can tell how many of these clicks actually results in a request. Would anyone be able to do that?

[Gemmamysoc]

There is a 'Making a request' goal set up already in Analytics that tracks who goes on to make requests. Is something further required? Or is it not set up quite right?

[MyfanwyNixon]

As best I understand it, a goal has to be labelled a conversion with Adwords tracking, not just analytics. But Nick knows more.

[garethrees]

Looks like we need to implement conversion tracking on RTK like we did on WDTK (also https://github.com/mysociety/whatdotheyknow-theme/issues/298). Will need to set up a conversion in Adwords and then RTK team will need to add the code to their theme.

[kat]

Thanks! @garethrees 👍 Will keep an eye on this thread for further instructions for RTK

[lizconlan]

Have added a Conversion to Adwords that mirrors the WDTK 'Made a Request' one and opened a PR on RTK to add the tracking code to the theme

https://github.com/openaustralia/righttoknow/pull/624

(Grudgingly from my own account as GitHub got confused about forking a renamed fork of a mySoc project)

[lizconlan]

Conversion tracking appears to be up and running. Will give it a day or so before removing the needs-config... label, just in case

[lizconlan]

there's still only 1 conversion, not sure whether this indicates a problem or not

[MyfanwyNixon]

It doesn't sound entirely unlikely, sadly :(

(How long is the conversion window btw? - If they see the site, think about it and come back two weeks later to make a request, would we know?)

[lizconlan]

The conversion window is set to the suggested default of 30 days

[MyfanwyNixon]

That seems reasonable.

[henare]

Hi everyone. As noted over on https://github.com/openaustralia/righttoknow/issues/549#issuecomment-251273298 @equivalentideas and I think we might have found a problem with the campaign setup and therefore the experiment. It seems the target location for these ads is the UK, not Australia. Surely this means that people are less likely to click or convert due to these ads.

Does this sound right to you?

[MyfanwyNixon]

Er.. no it doesn't sound right! Good spot ... Possibly it's a default from them having been set up at this end. Are you able to change them?

[kat]

done.

kat@oaf.org.au Founder, Director, Worker OpenAustralia Foundation http://openaustraliafoundation.org.au/ openaustralia http://openaustralia.org/ | planningalerts http://planningalerts.org.au/| electionleaflets http://electionleaflets.org.au/ | righttoknow http://righttoknow.org.au/ | theyvoteforyou http://theyvoteforyou.org.au/

On 4 October 2016 at 19:09, MyfanwyNixon notifications@github.com wrote:

Er.. no it doesn't sound right! Good spot ... Possibly it's a default from them having been set up at this end. Are you able to change them?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mysociety/alaveteli-experiments/issues/48#issuecomment-251322247, or mute the thread https://github.com/notifications/unsubscribe-auth/AAF0kFYAKJToN2IpWpsm03xMFsk9rf2wks5qwgnVgaJpZM4IXTNu .

[Gemmamysoc]

Hey guys - have you noticed any results on this experiment yet? I'll take a look, but wondered if you'd noticed anything during your Adwords exploration :)

[henare]

Hey @Gemmamysoc, I just checked. Prior to the targeting change there were 6153 clicks and 0 conversions. Since fixing that up we've had 691 clicks and 0 conversions for making a request (but 1 report a request conversion and 1 follow a request conversion).

I notice the most common ad served (37% of the time) is one that goes to the righttoknow.org.au homepage, not a body page.

I think this means in those circumstances it's not really testing "Can we use the PublicBody#notes attribute to improve Adwords traffic?" because the user isn't seeing that message when they click the ad. What do you think?

[Gemmamysoc]

Thanks @henare. Hmm, yeh...it seems we've deviated too much away from the purpose of the original experiment here I think. I'll discuss possible next steps with the team here!

[equivalentideas]

@gemma one side impact you might be interested in is that when we saw how much traffic we gained while the Universities experiment was on (even though people weren't making requests made us think we should be paying more attention to AdWords. That's what prompted us to start our current project on it https://github.com/openaustralia/righttoknow/issues/549#issuecomment-262408017

[Gemmamysoc]

Ah, yes, that makes sense :)

[Gemmamysoc]

Summary of what we've learnt from this experiment (there is more detailed analysis on this document):

• Adding more detailed notes to university body pages' description had a (small) bump in organic search traffic

• When we had the Google Adwords ads targeted at the UK, we had a lot of clicks, which reduced when we retargeted at Australia, see:

• However, the biggest boost to traffic to the universities included in the Adwords ads came after retargeting to Australia, so the Adwords had a good impact in terms of traffic, see:

• The Adwords ads that mentioned a specific university (so ANU, Sydney and Melbourne) had a higher total number of clicks (436) than the Adwords ads that didn't specifically mention a university and that were more generic (394 clicks). This implies that writing more specific Adwords ads about individual authorities is more effective in bringing traffic.

• “Entry requirements” is the most used search term and was the term used on most popular Adwords ads. This implies that individuals are searching for information that could help them directly (as opposed to e.g. “statistics” which sounds more like a researcher’s question)

• We don’t use “entry requirements” in the notes fields on university public body pages (we use "As a university, the University of [X] will hold information on its admission statistics and policy.") - so it might be a good idea to change the wording of these to include "entry requirements" specifically, considering the above discovery.

[Gemmamysoc]

Possible next steps:

1. Change the wording on half of the university body pages notes sections from "As a university, the University of [X] will hold information on its admission statistics and policy." to "As a university, the University of [X] will hold information on its entry requirements". Then 3-6 months later we should look to see if the organic search from Google is higher amongst these bodies after this wording change.

2. @henare @kat @equivalentideas You guys should feel free to stop running the 'RightToKnow Experiments' Adwords ads now if you'd prefer to use your budget elsewhere. Alternatively, we could think of a way to do a similar experiment to this but with different public bodies. We don't think it's the optimum time to do further testing on university bodies at the moment at it seems like the busiest period for students applying to universities in Australia might be over?