Retrieve videos from instagram hashtag function

[Tyrannas]

Video links are retrieved when scrapping user profiles on instagram, but not when searching with a hashtag, ony the cover images are retrieved. (Or it seems so actually).

Is it possible and I didnt find the option? If it's just not implemented but possible maybe I can try to contribute if maybe you can show me where to look.

Thanks

[Yomguithereal]

Is it possible and I didnt find the option?

There is no option for this indeed.

If it's just not implemented but possible maybe I can try to contribute if maybe you can show me where to look.

It's possibly not implemented indeed. Worst case scenario might be this info is collected through another API call by the website and we don't have it readily available in the parsed payloads.

You need to log the payload in this function here: https://github.com/medialab/minet/blob/master/minet/instagram/api_scraper.py#L381

Then the dataclass representing a post is there: https://github.com/medialab/minet/blob/master/minet/instagram/types.py#L84 and this is the method used to convert the payload into the beforementioned dataclass: https://github.com/medialab/minet/blob/master/minet/instagram/types.py#L104

I'll try to give it a look later if you don't find the time to look on your end.

[Tyrannas]

Okey thanks! 'm currently debugging but there does not seem to be a video url indeed, only images. So it's probably another API call, I will try to dig a bit.

[Tyrannas]

Okey so I found how to retrieve the url, so a simple reproducing step would be like that. But I don't know if it's the way you want it integrated, and how you want to integrate this new field in the output.

I guess the datamodel of InstagramHashtagPost should be modified ton include video_url, and then the csv output too, I can look it up if you give me some guidelines for proper contributing. Or I can let you do it.

But I need to mention that making the request seems to be super slow so maybe it's better if it's optionnal. And maybe the graphql can accept multiple shortcodes at once ? It would speed things up a lot.

for edge in edges:
    post = InstagramHashtagPost.from_payload(edge["node"])
    if post.media_type == "GraphVideo":
        url = forge_post_url_from_shortcode(post.shortcode)
        data = self.request_json(url)
        post.video_url = data['graphql']['shortcode_media']['video_url']
    yield post

[Yomguithereal]

In this case, since we are using the same routine as the post-infos command I think it would be better to split the workflow in two using the existing commands no? 1. you collect the posts using the hashtag command 2. you use the post-infos command on the relevant column of the output from step 1.

And maybe the graphql can accept multiple shortcodes at once ?

Maybe, I have never tested this. If you find it to be the case, tell me and we will improve the related command.

[Tyrannas]

Hmmm so. I tried minet instagram post-infos shortcode -i res.csv but i'm getting a File "d:\alie\documents\projets\minet\minet\instagram\types.py", line 171, in from_payload media_type = INSTAGRAM_MEDIA_TYPE.get(payload["media_type"]) error, I tried debugging it and indeed minet tries to access InstagramPost.from_payload(getpath(data, ["items", 0])) and "items" does not exist in the data dict, I guess it should be ["graphql", "shortcode_media"] from what I'm seeing. But even like this some required fields like media_type don't exist. So maybe the result from the request has changed since the code was written.

I also tried minet instagram post-infos id -i res.csv > output.csv but there i'm only getting raw html in stdout and no error nor output in output.csv (except for the header).

If the requests have changed, maybe it could be intersting to defer the calls to a dedicated lib like https://github.com/subzeroid/instagrapi ? Don't know if it's feasible or not, but it seems to be up to date.

But again, maybe i'm just mistaken and using minet the wrong way.

[Yomguithereal]

Hmm.

On my end minet instagram post-infos shortcode -i res.csv works. Can you give me a shortcode that fails? Best case scenario it is a simple fix.

If the requests have changed, maybe it could be intersting to defer the calls to a dedicated lib like https://github.com/subzeroid/instagrapi ? Don't know if it's feasible or not, but it seems to be up to date.

Looks like a good inspiration indeed.

[Tyrannas]

I just tried again and I get the same error with minet instagram post-infos shortcode -i res.csv > output.csv Here is my res.csv (it's only 5 rows) res.csv

[Yomguithereal]

@Tyrannas I failed to reproduce using your file. Can you double check your minet version?

[Tyrannas]

I'm on 2.0.3, using dev version (cloned the repo and pip installed -e), I tried to pull but apparently everything is up to date. I'll try on another computer though.