search

cncf / landscapeapp

🌄Upstream landscape generation application
https://landscapes.dev/
Apache License 2.0
253 stars 124 forks source link

ASWF tweet count is wrong #344

Closed dankohn closed 4 years ago

dankohn commented 4 years ago

The ASWF tweet count shows 37 but should show 1.

https://landscape.aswf.io https://twitter.com/search?q=https%3A%2F%2Flandscape.aswf.io&src=typd

It is probably because we copied the LF Edge website to start it.

jordinl commented 4 years ago

@dankohn this count seems to be based on a search performed on AcademySwf.

dankohn commented 4 years ago

It should instead be a search for landscape.aswf.io.

dankohn commented 4 years ago

It should be using: https://github.com/AcademySoftwareFoundation/aswf-landscape/blob/master/settings.yml#L21

jordinl commented 4 years ago

@dankohn actually, it seems to be using twitter.search from settings. But what the search is doing is fetching tweets since the last id and adding the result to the existing count. So if the count is wrong, we can either update processed_landscape.yml to have count = 1 or we will have to find an alternative way of counting tweets.

I see we're using the Standard Search API, which has a 7-day archive limit, so there's no way to count all tweets with that.

But the Premium Search API has access to the full-archive and it has a counts endpoint:

https://developer.twitter.com/en/docs/tweets/search/overview

I wonder if we would be able to switch to the Premium plan.

dankohn commented 4 years ago

This is by design. We didn't think it was worth paying for a premium Twitter license as we can just compare the historic count stored in processed_landscape.yml with the count since last update.

What we didn't plan for was the proliferation of landscapes. Could you please create a script that zeroes out the Twitter count and puts in the current timestamp, which I can run once on each new landscape so that the count will start from zero?

Dan Kohn dan@linuxfoundation.org +1-415-233-1000 Executive Director, Cloud Native Computing Foundation cncf.io https://www.cncf.io/ dankohn.com https://www.dankohn.com/ or book on my calendar: dankohn.com/c https://www.dankohn.com/c

On Tue, Sep 24, 2019 at 2:15 AM Jordi Noguera notifications@github.com wrote:

@dankohn https://github.com/dankohn actually, it seems to be using twitter.search from settings. But what the search is doing is fetching tweets since the last id https://github.com/AcademySoftwareFoundation/aswf-landscape/blob/master/processed_landscape.yml#L3479 and adding the result to the existing count https://github.com/AcademySoftwareFoundation/aswf-landscape/blob/master/processed_landscape.yml#L3479. So if the count is wrong, we can either update processed_landscape.yml to have count = 1 or we will have to find an alternative way of counting tweets.

I see we're using the Standard Search API, which has a 7-day archive limit, so there's no way to count all tweets with that.

But the Premium Search API has access to the full-archive and it has a counts endpoint:

https://developer.twitter.com/en/docs/tweets/search/overview

I wonder if we would be able to switch to the Premium plan.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cncf/landscapeapp/issues/344?email_source=notifications&email_token=AAXQYBR4TWV3NALL24QLV33QLGWADA5CNFSM4IXGXUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7NGMMY#issuecomment-534406707, or mute the thread https://github.com/notifications/unsubscribe-auth/AAXQYBT55F5KFUZSST4T2O3QLGWADANCNFSM4IXGXUTA .

jordinl commented 4 years ago

@dankohn I can totally add a script to reset the tweet count to 0. But if the use case is when creating a new landscape, wouldn't it make more sense to not copy processed_landscape.yml?

dankohn commented 4 years ago

We could, but I think a script would be clearer. Processed landscape can be really slow to regenerate from scratch.