Getting 403s and 404s on tvtvus and movistarplus

[Mauricio300808]

Site

tvtv.us and movistarplus.es

Description

Getting 403s and 404s on tvtvus and movistarplus

[Guito-del]

Hi, movistarplus.es is broken. Any update is available ?? Thx

[nickleby]

tvtv.us seems to only need headers to avoid the 403, movistarplus changed the epg schedule website and now the structure is very similar to the andorradifusio.ad site, maybe @freearhey can take a look at it and adapt the andorradifusio.ad to movistarplus.es :)

[davidclaeysquinones]

I've been investigating the issue for movistar plus and they recently changed their api. UP until this point I haven't found a way to fix the issue.

[davidclaeysquinones]

look at https://github.com/iptv-org/epg/pull/2440

[fraudiay79]

I'm currently working on an alternative to get the guide from orangetv.orange.es (https://orangetv.orange.es/epg), but with my limited knowledge in js scripting I have run into a stumbling block. I'm able to parse the channels correctly, but the epg content returns 0 programs. Can someone here take a look and correct it; then share with this repo? Thank you.

const dayjs = require('dayjs')

module.exports = {
  site: 'orangetv.es',
  days: 2,
  request: {
    cache: {
      ttl: 60 * 60 * 1000 // 1 hour
    }
  },
  url({ date }) {
    return `https://epg.orangetv.orange.es/epg/Smartphone_Android/1_PRO/${date.format('YYYYMMDD')}_8h_1.json`
  },
  parser: function ({ content }) {
    let programs = []
    const items = parseItems(content)
    items.forEach(item => {
      programs.push({
        title: item.programs.name,
        description: item.programs.description,
        season: item.programs.seriesSeason || null,
        episode: item.programs.episodeId || null,
        start: parseStart(item),
        stop: parseStop(item)
      })
    })

    return programs
  },
  async channels() {
    const axios = require('axios')
    const data = await axios
      .get(`https://pc.orangetv.orange.es/pc/api/rtv/v1/GetChannelList?bouquet_id=1&model_external_id=PC&filter_unsupported_channels=false&client=json`)
      .then(r => r.data)
      .catch(console.log)
    return data.response.map(item => {
      return {
        lang: 'es',
    name: item.name,
        site_id: item.externalChannelId
      }
    })
  }
}

function parseStart(item) {
  if (!item.programs || !item.programs.startDate) return null

  return item.startDate ? dayjs.unix(item.programs.startDate) : null
}

function parseStop(item) {
  if (!item.programs || !item.programs.endDate) return null

  return item.endDate ? dayjs.unix(item.programs.endDate) : null
}

function parseItems(content) {
  const data = JSON.parse(content)
  if (!data || !data.programs) return []

  return data.programs
}

[davidclaeysquinones]

The PR is ready and modified, so not necessarily needed to make code for a new provider.

I'll indulge myself and point out the existence of https://github.com/davidclaeysquinones/epg-info-docker. With the custom fixes enabled all should work again.

[fraudiay79]

Awesome. Would you have time to take a look at my file as well as a backup source?

[davidclaeysquinones]

I'm more than happy to look at it over the weekend. Another provider certainly can't hurt. Just in order to set my expectations, have you already tested your code ?

[Mauricio300808]

Nice!! I also pushed a change to add headers to tvtv.us and seems to be working fine. Thank you guys

[fraudiay79]

Thank you. I ran separate tests for retrieving the channels list and the epg. When I tested the generating the channels it was successful. When I try to grab the epg content it retrieves 0 programs for each channel, and no error is displayed

[Mauricio300808]

@davidclaeysquinones i tried the code in your PR but still getting 0 channels from movistar :/ I used this code in the config file

`const { DateTime } = require('luxon');

const API_CHANNEL_ENDPOINT = 'https://www.movistarplus.es/programacion-tv';
const API_PROGRAM_ENDPOINT = 'https://comunicacion.movistarplus.es';
const API_IMAGE_ENDPOINT = 'https://www.movistarplus.es/recorte/n/caratulaH/';

module.exports = {
  site: 'movistarplus.es',
  days: 2,
  url: function ({ date, channel }) {
    if (channel) {
      return `${API_PROGRAM_ENDPOINT}/wp-admin/admin-ajax.php`;
    }
    return `https://www.movistarplus.es/programacion-tv/${date.format('YYYY-MM-DD')}?v=json`;
  },
  request: {
    method: 'POST',
    headers: {
      Origin: API_PROGRAM_ENDPOINT,
      Referer: `${API_PROGRAM_ENDPOINT}/programacion/`,
      "Content-Type": 'application/x-www-form-urlencoded; charset=UTF-8',
    },
    data: function ({ channel, date }) {
      return {
        action: 'getProgramation',
        day: date.format('YYYY-MM-DD'),
        "channels[]": channel.site_id,
      };
    },
  },
  parser({ content, channel, date }) {
    let programs = [];
    let items = parseItems(content, channel);
    if (!items.length) return programs;

    items.forEach((item) => {
      let startTime = DateTime.fromFormat(
        `${date.format('YYYY-MM-DD')} ${item.HORA_INICIO}`,
        'yyyy-MM-dd HH:mm',
        {
          zone: 'Europe/Madrid',
        }
      ).toUTC();

      let stopTime = DateTime.fromFormat(
        `${date.format('YYYY-MM-DD')} ${item.HORA_FIN}`,
        'yyyy-MM-dd HH:mm',
        {
          zone: 'Europe/Madrid',
        }
      ).toUTC();

      if (stopTime < startTime) {
        stopTime = stopTime.plus({ days: 1 });
      }

      programs.push({
        title: item.des_evento_rejilla,
        category: item.des_genero,
        icon: parseIcon(item, channel),
        start: startTime,
        stop: stopTime,
      });
    });

    return programs;
  },
};

function parseIcon(item, channel) {
  return `${API_IMAGE_ENDPOINT}/M${channel.site_id}P${item.ELEMENTO}`;
}

function parseItems(content, channel) {
  const json = typeof content === 'string' ? JSON.parse(content) : content;
  const data = json.channelsProgram;

  if (!data || data.length !== 1) return [];
  return data[0].PROGRAMAS || [];
}
`

[davidclaeysquinones]

I hope they haven't changed the API again When I tested it worked.

[Mauricio300808]

@davidclaeysquinones this code works for me:

const { DateTime } = require('luxon');
const axios = require('axios');

const API_PROGRAM_ENDPOINT = 'https://comunicacion.movistarplus.es';
const API_IMAGE_ENDPOINT = 'https://www.movistarplus.es/recorte/n/caratulaH/'; // Define the image endpoint

module.exports = {
  site: 'movistarplus.es',
  days: 2,
  url: function ({ channel, date }) {
    const luxonDate = DateTime.fromJSDate(new Date(date)); // Ensure `date` is converted
    return `${API_PROGRAM_ENDPOINT}/wp-admin/admin-ajax.php`;
  },
  request: {
    method: 'POST', // Try GET if POST fails
    headers: {
      Origin: API_PROGRAM_ENDPOINT,
      Referer: `${API_PROGRAM_ENDPOINT}/programacion/`,
      "Content-Type": 'application/x-www-form-urlencoded; charset=UTF-8'
    },
    data: function ({ channel, date }) {
      const luxonDate = DateTime.fromJSDate(new Date(date)); // Ensure `date` is converted
      return {
        action: 'getProgramation',
        day: luxonDate.toFormat('yyyy-MM-dd'),
        "channels[]": channel.site_id
      };
    }
  },
  async fetchData({ channel, date }) {
    const luxonDate = DateTime.fromJSDate(new Date(date)); // Ensure `date` is converted
    const payload = {
      action: 'getProgramation',
      day: luxonDate.toFormat('yyyy-MM-dd'),
      "channels[]": channel.site_id
    };

    try {
      console.log('Request Payload:', payload);

      const response = await axios({
        method: this.request.method,
        url: this.url({ channel, date }),
        headers: this.request.headers,
        data: new URLSearchParams(payload).toString() // For POST
      });

      console.log('Response Status:', response.status);
      console.log('Response Data:', response.data);

      return this.parser({ content: response.data, channel, date });
    } catch (error) {
      console.error('Error Status:', error.response?.status);
      console.error('Error Data:', error.response?.data);

      // If 405, retry with GET method
      if (error.response?.status === 405 && this.request.method === 'POST') {
        console.log('Retrying with GET...');
        return this.retryWithGet({ channel, date });
      }

      return null; // Handle the error gracefully
    }
  },
  async retryWithGet({ channel, date }) {
    const luxonDate = DateTime.fromJSDate(new Date(date));
    const query = new URLSearchParams({
      action: 'getProgramation',
      day: luxonDate.toFormat('yyyy-MM-dd'),
      "channels[]": channel.site_id
    }).toString();

    const url = `${this.url({ channel, date })}?${query}`;
    try {
      const response = await axios.get(url, { headers: this.request.headers });
      console.log('GET Response Status:', response.status);
      console.log('GET Response Data:', response.data);
      return this.parser({ content: response.data, channel, date });
    } catch (error) {
      console.error('GET Error Status:', error.response?.status);
      console.error('GET Error Data:', error.response?.data);
      return null;
    }
  },
  parser({ content, channel, date }) {
    const json = typeof content === 'string' ? JSON.parse(content) : content;

    // Ensure proper parsing of the data
    if (!json || !json.channelsProgram || json.channelsProgram.length === 0) {
      console.error('No programs found in the response.');
      return [];
    }

    const programs = json.channelsProgram[0]; // Assuming first array contains program data

    // Map over the programs and parse relevant data
    return programs.map((item) => {
      const startTime = DateTime.fromFormat(
        `${date} ${item.f_evento_rejilla}`,
        'yyyy-MM-dd HH:mm:ss',
        { zone: 'Europe/Madrid' }
      ).toUTC();

      const stopTime = DateTime.fromFormat(
        `${date} ${item.f_fin_evento_rejilla}`,
        'yyyy-MM-dd HH:mm:ss',
        { zone: 'Europe/Madrid' }
      ).toUTC();

      return {
        title: item.des_evento_rejilla,
        icon: this.parseIcon(item), // Use the `parseIcon` function
        category: item.des_genero,
        start: startTime,
        stop: stopTime
      };
    });
  },
  parseIcon(item) {
    // Construct the icon URL
    return `${API_IMAGE_ENDPOINT}${item.cod_evento_rejilla}.jpg`; // Ensure it's referencing the correct image
  }
};

[davidclaeysquinones]

Did you get the images to work properly?

[Guito-del]

This Code worked for me too. THX all

[Mauricio300808]

here is a much cleaner code. I do not see the icons though...

const { DateTime } = require('luxon');

const API_CHANNEL_ENDPOINT = 'https://www.movistarplus.es/programacion-tv';
const API_PROGRAM_ENDPOINT = 'https://comunicacion.movistarplus.es';
const API_IMAGE_ENDPOINT = 'https://www.movistarplus.es/recorte/n/caratulaH/';

module.exports = {
  site: 'movistarplus.es',
  days: 2,
  url: function ({ channel, date }) {
    return `${API_PROGRAM_ENDPOINT}/wp-admin/admin-ajax.php`;
  },
  request: {
    method: 'POST',
    headers: {
      Origin: API_PROGRAM_ENDPOINT,
      Referer: `${API_PROGRAM_ENDPOINT}/programacion/`,
      "Content-Type" : 'application/x-www-form-urlencoded; charset=UTF-8',
    },
    data: function ({ channel, date }) {
      return {
        action: 'getProgramation',
        day: date.format('YYYY-MM-DD'),
        "channels[]": channel.site_id,
      };
    },
  },
  parser({ content, channel, date }) {
    let programs = [];
    let items = parseItems(content, channel);
    if (!items.length) return programs;

    items.forEach(item => {
      let startTime = DateTime.fromFormat(
        `${item.f_evento_rejilla}`,
        'yyyy-MM-dd HH:mm:ss',
        { zone: 'Europe/Madrid' }
      ).toUTC();

      let stopTime = DateTime.fromFormat(
        `${item.f_fin_evento_rejilla}`,
        'yyyy-MM-dd HH:mm:ss',
        { zone: 'Europe/Madrid' }
      ).toUTC();

      // Adjust stop time if it's on the next day
      if (stopTime < startTime) {
        stopTime = stopTime.plus({ days: 1 });
      }

      programs.push({
        title: item.des_evento_rejilla,
        icon: parseIcon(item, channel),
        category: item.des_genero,
        start: startTime,
        stop: stopTime,
      });
    });

    return programs;
  },
};

function parseIcon(item, channel) {
  return `${API_IMAGE_ENDPOINT}/M${channel.site_id}P${item.ELEMENTO}`;
}

function parseItems(content, channel) {
  const json = typeof content === 'string' ? JSON.parse(content) : content;
  const data = json.channelsProgram;

  if (data.length !== 1) return [];
  return data[0];
}

[davidclaeysquinones]

Well in the first version you used item.cod_evento_rejilla and in the second item.ELEMENTO. I was having trouble retrieving the program images with the new API version so I was wondering if you figured that out.

[davidclaeysquinones]

I've updated https://github.com/iptv-org/epg/pull/2440. It now also gets the correct program images.

In order it get it mergeable I need to rework the test suite for the provider. If someone wants to help out feel free to contribute to https://github.com/davidclaeysquinones/epg/tree/movistar.es.

Until the PR is not approved the fixes won't be available on the main branch. First it needs to get in a mergeable state and after that it depends on when a maintainer approves it.

Up until then you can edit the server code manually or if you use the Docker image you can enable the custom fixes.

[davidclaeysquinones]

@fraudiay79 I haven't got around checking your code. If you're okay with at some point I will make a branch on my fork so that at some point we can make a PR. I can add you as a contributor so that you can take credit of your work.

[davidclaeysquinones]

@fraudiay79 I took a look at your code and after some tweaking I manages to get something going. You can take a look at my fork of the repo. My plan is to at some point submit a PR, but before that I need to write some tests.