Fix MacOS build due to failing dependency installations

[andrewtavis]

Terms

• [X] I have searched all open bug reports
• [X] I agree to follow Scribe-Data's Code of Conduct

Behavior

As of now the build for all versions of MacOS is failing. This was originally because of Tensorflow, which has now been removed as a dependency in favor of other packages that were also being used and had some similar functionality (parallel downloads). The most recent error is:

ERROR: Could not build wheels for marisa-trie, which is required to install pyproject.toml-based projects
Successfully built PyICU sentencepiece
Failed to build marisa-trie

Would be great if we could figure this out!

[andrewtavis]

CC @wkyoshida, in case you have some ideas on this :)

[shashank-iitbhu]

The workflow is failing because one of the dependancies language-data depends on marisa-trie<0.8.0 and >=0.7.7. And marisa-trie<0.8.0 and >=0.7.7 does not have support for Python 3.12 which is the error in the CI logs. As mentioned in the marisa-trie docs, marisa-trie 1.1.0 has added support for python 3.12 .

The solution is to remove the language-data dependancy from requirements.txt as I think it is not being used anywhere in the project.

I also tried pinning both language-data and marisa-trie to their latest versions but this won't work beacuse of this:

ERROR: Cannot install language-data==1.1 and marisa-trie>=1.1.0 because these package versions have conflicting dependencies.

    The user requested marisa-trie>=1.1.0
    language-data 1.1 depends on marisa-trie<0.8.0 and >=0.7.7

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts```

[shashank-iitbhu]

@andrewtavis @wkyoshida

[andrewtavis]

Thanks for looking into this, @shashank-iitbhu! 😊 I just checked and the following is in the pyproject.toml for langcodes:

[tool.poetry.dependencies]
python = ">= 3.6"
language-data = { version = "^1.1", optional = true }

[tool.poetry.dev-dependencies]
language-data = { version = "^1.1", optional = true }

[tool.poetry.extras]
data = ["language-data"]

Do you want to give try to remove language-data from the dependencies and hopefully it'll work, @shashank-iitbhu? We can also just not support Python 3.12 right now. It seems like there was a recent fork of language-data that's now maintained :)

[andrewtavis]

It looks like the new language-data will be just the lower bound of marisa-trie, but then it seems to be already in production. Let's give unpinning a try and focus on testing Python 3.9 as we are now.

[andrewtavis]

Thinking about this a bit more and what a hassle langcodes/language-data and marisa-trie have been since being added in, would it make sense to revert this and shift back to a dictionary that we're maintaining based on the languages we have the in extract_transform/languages directory? We just wanted this for language to ISO-2 conversion.

CC @wkyoshida

[andrewtavis]

@shashank-iitbhu, would you want to implement a reverse of #55 such that we have the codes saved for all the current languages in the languages directory? We'd be adding those in to language_meta_data.json :)

[shashank-iitbhu]

Sure @andrewtavis, will open a PR for this.

[andrewtavis]

Thanks, @shashank-iitbhu!

[andrewtavis]

Closed by #129. Thanks for this, @shashank-iitbhu!

[andrewtavis]

Reopening here and removing the mac build for now as this is once again problematic...

[andrewtavis]

Well would you look at them Mac builds! The combination of the following got this fixed:

• 71b4800
• a640948
• a82b9f1

I referenced the following Stack Overflow answer when I was setting up the project again from scratch and realized that it likely could help with the problems we were having here 😊

Closing until the next time the Mac builds break! 😅