No module named pygmo

[jack89roberts]

@jack89roberts perfect! LGTM 👍 Nice work :)

Something to keep in mind however, running airsenal_run_pipeline --weeks_ahead 3 will produce ModuleNotFoundError: No module named 'pygmo' error. Not sure if this is because the season hasn't started yet and pipeline uses airsenal_run_optimization instead of airsenal_make_squad?

Originally posted by @subash774 in https://github.com/alan-turing-institute/AIrsenal/issues/491#issuecomment-1192378145

[jack89roberts]

Thanks for flagging this @subash774 , pygmo should be an optional dependency so the code should run without it - we'll need to check the logic for checking whether it's present or not.

The only way to install it on Mac (or Windows) is with conda or using docker (that's why we have it as optional). So if you'd like to run it before we get to fixing the bug you can install AIrsenal in a conda environment (see the instructions in the readme). Pygmo will also give better results for optimising initial squads (it's only used by airsenal_make_squad, not airsenal_run_optimization).

[subash774]

Hey, run_make_squad in airsenal_run_pipeline.py uses pygmo however the function in squad_builder.py seems to have a try catch block to use "normal" algorithm instead, is it worth making a new invocation script (something like initialise_squad_for_new_season) for start of the season that uses fill_initial_squad in the pipeline instead?

I may have misunderstood the logic here, as the sequence of airsenal_setup_initial_db > airsenal_check_data > airsenal_update_db > airsenal_run_prediction --weeks_ahead 3 > airsenal_make_squad --num_gameweeks 3 works absolutely fine without pygmo, do we need pygmo specifically for airsenal_run_optimization --weeks_ahead 3? What if we just use the logic in fill_initial_squad instead? Or is that too simple as it doesn't take into account the substitutions (we'd need optimisation - permutation logic there right?)

Refs: https://github.com/alan-turing-institute/AIrsenal/blob/1ec68b33d7ddfa1fcc40343db852e1b52e7fb7df/airsenal/scripts/airsenal_run_pipeline.py#L214

https://github.com/alan-turing-institute/AIrsenal/blob/1ec68b33d7ddfa1fcc40343db852e1b52e7fb7df/airsenal/scripts/squad_builder.py#L23

[jack89roberts]

Yes you're right, run_make_squad in airsenal_run_pipeline should call squad_builder:fill_initial_squad rather than optimization_pygmo:make_new_squad_pygmo (and the pipeline script wouldn't need to call fill_initial_suggestion_table etc. after that either as that happens in fill_initial_squad). Pull requests welcome if you'd like to make the fix!

[subash774]

Coolio, I will get that done this weekend 👍

[subash774]

@jack89roberts, as the season hasn't yet started, I can't seem to test if run_optimization function works (I get transaction database error which I am guessing is expected at this point in time?), is there a fixture / mock that tests this?

[jack89roberts]

I don't think we do, but we have tests for most of the underlying key functions (if you run pytest). run_optimization should fail nicely and give a message to run airsenal_make_squad instead before the start of the season, though, so that's another bug. Could you post the error message?

It is possible to run the optimisation on previous seasons' data if you like (see the --help for each command, or try with airsenal_replay_season).

[subash774]

When you run optimization, it simply raises ValueError as below. I will see if we can raise error informing to run make squad instead if running for the first time 👍

Running optimization with fpl_team_id 1634671
  0%|                                                                                                                                     | 0/100 [00:00<?, ?it/sG
etting starting squad for 1634671                                                                                                         | 0/100 [00:00<?, ?it/s]
Traceback (most recent call last):                                                                                                        | 0/100 [00:00<?, ?it/s]
  File "/Users/subash774/Projects/test_fpl/AIrsenal/venv/bin/airsenal_run_optimization", line 8, in <module>                              | 0/100 [00:00<?, ?it/s]
    sys.exit(main())|                                                                                                                      | 0/21 [00:00<?, ?it/s]
  File "/Users/subash774/Projects/test_fpl/AIrsenal/airsenal/scripts/fill_transfersuggestion_table.py", line 764, in main
    run_optimization(
  File "/Users/subash774/Projects/test_fpl/AIrsenal/airsenal/scripts/fill_transfersuggestion_table.py", line 482, in run_optimization
    starting_squad = get_starting_squad(
  File "/Users/subash774/Projects/test_fpl/AIrsenal/airsenal/framework/optimization_utils.py", line 146, in get_starting_squad
    raise ValueError(f"No transactions in database for team ID {fpl_team_id}")

[JPKFin]

FWIW, I'm getting a bit different error on run_optimization when running on my M1:

airsenal_run_optimization --weeks_ahead 3
Logged in successfully
Running optimization with fpl_team_id 3170725
Unable to access FPL API
Traceback (most recent call last):
  File "/Users/JPKFin/Premier2223/AIrsenal/venv/bin/airsenal_run_optimization", line 8, in <module>
    sys.exit(main())
  File "/Users/JPKFin/Premier2223/AIrsenal/venv/lib/python3.9/site-packages/airsenal/scripts/fill_transfersuggestion_table.py", line 764, in main
    run_optimization(
  File "/Users/JPKFin/Premier2223/AIrsenal/venv/lib/python3.9/site-packages/airsenal/scripts/fill_transfersuggestion_table.py", line 421, in run_optimization
    num_free_transfers = get_free_transfers(
  File "/Users/JPKFin/Premier2223/AIrsenal/venv/lib/python3.9/site-packages/airsenal/framework/utils.py", line 316, in get_free_transfers
    return apifetcher.get_num_free_transfers(fpl_team_id)
  File "/Users/JPKFin/Premier2223/AIrsenal/venv/lib/python3.9/site-packages/airsenal/framework/data_fetcher.py", line 207, in get_num_free_transfers
    return squad_data["transfers"]["limit"]
TypeError: 'NoneType' object is not subscriptable

However the API seems to work if I try it directly python Python 3.9.6 (default, Jun 28 2021, 19:24:41) [Clang 12.0.5 (clang-1205.0.22.9)] on darwin Type "help", "copyright", "credits" or "license" for more information.

from airsenal.framework.utils import get_player p = get_player(35) p.name 'Robin Olsen' quit()

[nbarlowATI]

Hi @JPKFin, I can reproduce this, and I think it's an actual bug that we introduced last season, when we added the "login" feature (in order to get the most accurate prices for bought/sold players) in the middle of the season, without being able to test what would happen before gameweek 1.
Hopefully there should be an easy(ish) fix - we will let you know!

[JPKFin]

Hi, I updated to the latest version from main (1.4.1) and tried again to: airsenal_run_optimization --weeks_ahead 3 This time with a different result (still on M1 mac):

Logged in successfully
Running optimization with fpl_team_id 3170725
Unable to access FPL API
No existing squad or transfers found for team_id 3170725
Will suggest a new starting squad:
No module named 'pygmo'
Defaulting to algorithm=normal instead
Choosing new squad: iteration 0
Choosing new squad: iteration 1

etc.. for 99 iterations after which a new squad is generated (pretty useless for me)

[subash774]

On M1 mac, it's working for me when I do export FPL_TEAM_ID=YOUR_ID && airsenal_update_db && airsenal_run_prediction --weeks_ahead 3 && airsenal_run_optimization --weeks_ahead 3

[JPKFin]

I was using the configuration file for the fpl_team_id (you can see from above that the app seems to be aware of my fpl_team_id). Now tried with both config file and environment variable but to no avail :( (actually I had a stupid mistake and was using last season's team id but no change even with this seasons team id) And it seems if I again try directly from python:

from airsenal.framework.utils import get_current_players
p = get_current_players()
p
[14, 318, 310, 161, 339, 350, 263, 157, 414, 114, 40, 295, 144, 364, 100]

which is indeed my current team..

[subash774]

This is all using poetry right? That's strange. My issue was #504 where it wasn't reading FPL team ID from file, using environment variable solved it all. Maybe try a fresh clone? 🤔 🤷

[JPKFin]

No, just running python in the Airsenal venv environment

Python 3.9.6 (default, Jun 28 2021, 19:24:41) 
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from airsenal.framework.utils import get_current_players
>>> p = get_current_players()
>>> p
[14, 325, 317, 162, 348, 359, 269, 158, 426, 115, 40, 301, 145, 374, 101]
>>> 

Interesting BTW that the player ID's changed when I made a fresh database withe airsenal_setup_inital_db --clean In the end of that procedure the team API strangely seems to work..

Getting initially selected players in squad 2893675 for first gameweek...
Got starting squad from gameweek 1. Adding player data...
DONE!

also the "transaction" table in the database has the right data with my 15 initial players.

[JPKFin]

@subash774: which python version, and which requests version are you running on - my python is on version (as shown above) 3.9.6 and my requests version is 2.28.1

[subash774]

Python version Python 3.9.13 and requests version 2.28.1. I use poetry so flow is something like

• git clone and cd AIrsenal
• poetry install
• poetry shell
• pip install .

[jack89roberts]

I hope to have time to look at and debug some of these tomorrow. But a few comments:

• The latest versions of AIrsenal should not give pygmo not found errors - if pygmo isn't available it should revert to using a simpler algorithm. If you're getting pygmo not found on the latest main branch that's a new bug.
• On Windows and Mac pygmo is only available with conda. So if you want to use it you'll need to use a conda environment (it's only used for creating full squads, not for the transfer optimisation). I think it's also only available up to Python 3.8. It's on our to-do list to find an alternative.
• The issue with not picking up FPL_TEAM_ID seems to be related to us moving to Poetry to manage our (non-pygmo) dependencies. But if you're logging into the API (to get the latest private status of your team rather than the latest public state) then make sure FPL_LOGIN and FPL_PASSWORD are defined as well.
• The bug where AIrsenal is running a squad generation instead of transfer optimisation is a weird one, it may be related to the login problems but we also made some tweaks to the logic for that last week.
• We also made some database changes in the most recent version, so it may be worth trying deleting the database (by default in /tmp/data.db) and then running the whole AIrsenal pipeline to recreate it.

[nbarlowATI]

I think the fact that it is running squad generation instead of transfer optimization is (since the latest version) a direct consequence of it not picking up FPL_TEAM_ID - since it can't find an existing squad (either via transfers in the database, or current squad though the API), it assumes it is a new FPL entry so needs to generate a new one.

So the base problem is the non-picking-up of FPL_TEAM_ID (and possibly FPL_LOGIN and FPL_PASSWORD). I also saw that last time I setup, for reasons I don't understand, these files were not copied from my local airsenal/data/ directory into the site-packages directory when I did pip install ..
So the (fairly unsatisfactory) workaround is to copy them by hand - if you do which python to find the path for your environment you should see something like /blah/blah/envs/airsenalenv/bin/python. You can then copy the FPL_TEAM_ID, FPL_LOGIN, and FPL_PASSWORD files to /blah/blah/envs/airsenalenv/lib/python3.9/site-packages/airsenal/data/ and it should work....

It would be really good for us to understand and fix this though!

[jack89roberts]

How about we move the config files to the home directory/similar? That would also ease the route to putting AIrsenal on PyPI in the future if we wanted to.

[JPKFin]

Problem solved. Did a fresh install with a new virtual environment once more, and decided to run the optimization script without submitting email and password and everything worked. Hooray! Then I ran the script again and submitted email and password when asked and let the script to store those. Everything worked again. Then checked the previous install and found out that I had a typo in the stored password :/ So it seems that if one provides an erroneous username or password, one gets first of all a bit confusing error messages and behaviour:

Running optimization with fpl_team_id 2893675
Unable to access FPL API
No existing squad or transfers found for team_id 2893675
Will suggest a new starting squad:
No module named 'pygmo'

Most likely the 'No module named 'pygmo'' comes from that "new starting squad" branch of the program ? (when everything worked, I did not get the pygmo-message)

Sorry for all the confusion caused by my typo..