search

pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
43.26k stars 17.8k forks source link

QST: read_html not continuous finding table when one exists #42779

Closed AndyYan108 closed 2 years ago

AndyYan108 commented 3 years ago

Question about pandas

Note: The program can run only 4 times in the for loop(like 1-5) and then the error comes out

#
import requests
import pandas as pd

header = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36'}

screen = requests.get('https://finviz.com/screener.ashx?v=111&f=cap_microunder,exch_nasd', headers = header).text

tables = pd.read_html(screen)

tables = tables[-2]

tables.columns = tables.iloc[0]

tables = tables[1:]

for i in range(1, 62):
    num = i * 20 + 1
    screen = requests.get(f'https://finviz.com/screener.ashx?v=111&f=cap_microunder,exch_nasd&r={num}', headers = header).text 
    tables1 = pd.read_html(screen)
    tables1 = tables1[-2]
    tables1.columns = tables1.iloc[0]
    tables1 = tables1[1:]
    tables = pd.concat([tables, tables1], ignore_index=True)
    print(tables)

tables.to_csv('table.csv')

Traceback

0  No. Ticker                               Company  ...  Price   Change    Volume
0    1   AACG                 ATA Creativity Global  ...   3.26    5.81%     74743
1    2   AAME         Atlantic American Corporation  ...   4.05   -0.74%     13064
2    3   AAOI         Applied Optoelectronics, Inc.  ...   7.50    4.02%    300084
3    4   AATC    Autoscope Technologies Corporation  ...   6.68    5.67%     23572
4    5   ABEO              Abeona Therapeutics Inc.  ...   1.24   -0.80%   1644640
5    6   ABGI               ABG Acquisition Corp. I  ...   9.73   -0.10%      1102
6    7   ABIO                  ARCA biopharma, Inc.  ...   3.02    1.34%    127806
7    8   ABUS         Arbutus Biopharma Corporation  ...   2.74    1.44%    557042
8    9   ACER                Acer Therapeutics Inc.  ...   2.40    1.69%     72867
9   10   ACET                      Adicet Bio, Inc.  ...   7.46   -0.40%     60257
10  11   ACEV     ACE Convergence Acquisition Corp.  ...   9.86    0.00%    227069
11  12   ACHL             Achilles Therapeutics plc  ...   6.02    0.17%      3129
12  13   ACHV           Achieve Life Sciences, Inc.  ...   7.26    0.12%    125431
13  14  ACKIU           Ackrell SPAC Partners I Co.  ...  10.21   -0.51%     20429
14  15   ACNB                      ACNB Corporation  ...  28.18   -0.04%      3847
15  16   ACOR             Acorda Therapeutics, Inc.  ...   3.89   -1.74%    314330
16  17   ACRX          AcelRx Pharmaceuticals, Inc.  ...   1.19    0.00%    491255
17  18   ACST                    Acasti Pharma Inc.  ...   0.47    3.67%   3735616
18  19   ACTG           Acacia Research Corporation  ...   5.72    0.88%     96306
19  20   ACXP           Acurx Pharmaceuticals, Inc.  ...   5.19    1.21%     72454
20  21   ADES    Advanced Emissions Solutions, Inc.  ...   7.64    2.41%     94783
21  22   ADIL           Adial Pharmaceuticals, Inc.  ...   2.38    2.14%    114676
22  23   ADMA                  ADMA Biologics, Inc.  ...   1.60   -3.03%    724842
23  24   ADMP    Adamis Pharmaceuticals Corporation  ...   1.03   -0.48%    770632
24  25   ADMS          Adamas Pharmaceuticals, Inc.  ...   4.86    1.25%     41785
25  26   ADOC                Edoc Acquisition Corp.  ...  10.05   -0.20%       101
26  27   ADTX                          Aditxt, Inc.  ...   2.15   -0.46%    245755
27  28   ADVM         Adverum Biotechnologies, Inc.  ...   2.32   -0.22%   1007877
28  29   ADXN                Addex Therapeutics Ltd  ...   9.00   -5.76%      1218
29  30   ADXS                         Advaxis, Inc.  ...   0.43    3.98%   1963730
30  31   AEAC    Authentic Equity Acquisition Corp.  ...   9.66   -0.21%         2
31  32   AEHL  Antelope Enterprise Holdings Limited  ...   2.93    4.08%    329894
32  33   AEHR                     Aehr Test Systems  ...   5.90    5.73%   1501288
33  34    AEI        Alset EHome International Inc.  ...   2.17  -36.22%  10833029
34  35   AEMD                 Aethlon Medical, Inc.  ...   5.31    9.98%    376869
35  36   AESE     Allied Esports Entertainment Inc.  ...   2.05    3.02%    150842
36  37    AEY   ADDvantage Technologies Group, Inc.  ...   2.23    0.00%      8484
37  38   AEYE                        AudioEye, Inc.  ...  13.95    3.68%     34198
38  39   AEZS                 Aeterna Zentaris Inc.  ...   0.77    1.25%   1025815
39  40   AFBI             Affinity Bancshares, Inc.  ...  13.24    0.00%      1592

[40 rows x 11 columns]
0  No. Ticker  ...   Change    Volume
0    1   AACG  ...    5.81%     74743
1    2   AAME  ...   -0.74%     13064
2    3   AAOI  ...    4.02%    300084
3    4   AATC  ...    5.67%     23572
4    5   ABEO  ...   -0.80%   1644640
5    6   ABGI  ...   -0.10%      1102
6    7   ABIO  ...    1.34%    127806
7    8   ABUS  ...    1.44%    557042
8    9   ACER  ...    1.69%     72867
9   10   ACET  ...   -0.40%     60257
10  11   ACEV  ...    0.00%    227069
11  12   ACHL  ...    0.17%      3129
12  13   ACHV  ...    0.12%    125431
13  14  ACKIU  ...   -0.51%     20429
14  15   ACNB  ...   -0.04%      3847
15  16   ACOR  ...   -1.74%    314330
16  17   ACRX  ...    0.00%    491255
17  18   ACST  ...    3.67%   3735616
18  19   ACTG  ...    0.88%     96306
19  20   ACXP  ...    1.21%     72454
20  21   ADES  ...    2.41%     94783
21  22   ADIL  ...    2.14%    114676
22  23   ADMA  ...   -3.03%    724842
23  24   ADMP  ...   -0.48%    770632
24  25   ADMS  ...    1.25%     41785
25  26   ADOC  ...   -0.20%       101
26  27   ADTX  ...   -0.46%    245755
27  28   ADVM  ...   -0.22%   1007877
28  29   ADXN  ...   -5.76%      1218
29  30   ADXS  ...    3.98%   1963730
30  31   AEAC  ...   -0.21%         2
31  32   AEHL  ...    4.08%    329894
32  33   AEHR  ...    5.73%   1501288
33  34    AEI  ...  -36.22%  10833029
34  35   AEMD  ...    9.98%    376869
35  36   AESE  ...    3.02%    150842
36  37    AEY  ...    0.00%      8484
37  38   AEYE  ...    3.68%     34198
38  39   AEZS  ...    1.25%   1025815
39  40   AFBI  ...    0.00%      1592
40  41   AFCG  ...    0.52%     15971
41  42   AGFS  ...    2.62%     52066
42  43   AGGR  ...   -0.10%     16350
43  44   AGMH  ...   -0.55%     69710
44  45   AGRI  ...    0.34%    249721
45  46   AGRX  ...    8.02%    502187
46  47   AGTC  ...    0.00%    194981
47  48   AHAC  ...   -0.21%     28010
48  49   AHPI  ...   -6.10%    594069
49  50    AIH  ...   11.89%    172789
50  51   AIHS  ...    0.28%    798443
51  52   AIKI  ...    1.85%    529251
52  53   AIRG  ...    2.19%     13836
53  54   AIRT  ...    2.38%      6816
54  55   AKIC  ...    0.00%         0
55  56   AKTX  ...    2.52%     69737
56  57   ALAC  ...   -0.09%      4935
57  58   ALCO  ...   -0.38%     15543
58  59    ALF  ...   -2.78%    900340
59  60   ALIM  ...   -0.69%      6687

[60 rows x 11 columns]
0  No. Ticker                              Company  ... Price  Change   Volume
0    1   AACG                ATA Creativity Global  ...  3.26   5.81%    74743
1    2   AAME        Atlantic American Corporation  ...  4.05  -0.74%    13064
2    3   AAOI        Applied Optoelectronics, Inc.  ...  7.50   4.02%   300084
3    4   AATC   Autoscope Technologies Corporation  ...  6.68   5.67%    23572
4    5   ABEO             Abeona Therapeutics Inc.  ...  1.24  -0.80%  1644640
..  ..    ...                                  ...  ...   ...     ...      ...
75  76   ANPC  AnPac Bio-Medical Science Co., Ltd.  ...  3.80   0.80%    33758
76  77   ANTE               AirNet Technology Inc.  ...  2.36   4.42%   237799
77  78    ANY                      Sphere 3D Corp.  ...  2.99   5.28%  2170342
78  79   APDN           Applied DNA Sciences, Inc.  ...  5.20  -1.70%   275280
79  80   APEN             Apollo Endosurgery, Inc.  ...  7.94   2.58%     8045

[80 rows x 11 columns]
0   No. Ticker                             Company  ...  Price  Change   Volume
0     1   AACG               ATA Creativity Global  ...   3.26   5.81%    74743
1     2   AAME       Atlantic American Corporation  ...   4.05  -0.74%    13064
2     3   AAOI       Applied Optoelectronics, Inc.  ...   7.50   4.02%   300084
3     4   AATC  Autoscope Technologies Corporation  ...   6.68   5.67%    23572
4     5   ABEO            Abeona Therapeutics Inc.  ...   1.24  -0.80%  1644640
..  ...    ...                                 ...  ...    ...     ...      ...
95   96   ARTL            Artelo Biosciences, Inc.  ...   1.02  -0.97%  2794669
96   97   ARTW   Art's-Way Manufacturing Co., Inc.  ...   3.60   2.27%    24353
97   98   ARYD   ARYA Sciences Acquisition Corp IV  ...  10.16  -0.26%     2014
98   99   ASLN       ASLAN Pharmaceuticals Limited  ...   2.87   6.30%   218557
99  100   ASMB          Assembly Biosciences, Inc.  ...   3.51   2.03%   206015

[100 rows x 11 columns]
Traceback (most recent call last):

  File "C:\Users\yanli\.spyder-py3\temp.py", line 20, in <module>
    tables1 = pd.read_html(screen)

  File "C:\Non-system\Anaconda\lib\site-packages\pandas\util\_decorators.py", line 296, in wrapper
    return func(*args, **kwargs)

  File "C:\Non-system\Anaconda\lib\site-packages\pandas\io\html.py", line 1086, in read_html
    return _parse(

  File "C:\Non-system\Anaconda\lib\site-packages\pandas\io\html.py", line 917, in _parse
    raise retained

  File "C:\Non-system\Anaconda\lib\site-packages\pandas\io\html.py", line 898, in _parse
    tables = p.parse_tables()

  File "C:\Non-system\Anaconda\lib\site-packages\pandas\io\html.py", line 217, in parse_tables
    tables = self._parse_tables(self._build_doc(), self.match, self.attrs)

  File "C:\Non-system\Anaconda\lib\site-packages\pandas\io\html.py", line 547, in _parse_tables
    raise ValueError("No tables found")

ValueError: No tables found
mroeschke commented 3 years ago

Could you post a more minimal example? https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

mzeitlin11 commented 2 years ago

Going to close for now, happy to reopen if you can provide a minimal example